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ANALYSIS  OF  ALTERNATIVES  (AO A)  PROCESS  IMPROVEMENT  STUDY 

SUMMARY 


THE  PROJECT  PURPOSE  was  to  develop  an  approach  to  focus  AoA  activities  to  most 
efficiently  and  effectively  deliver  results  that  meet  decision-maker  needs. 

THE  PROJECT  SPONSOR  was  the  Deputy  Chief  of  Staff,  G-8. 

THE  PROJECT  OBJECTIVES  were  to: 

(1)  Lay  out  all  the  analysis-related  AoA  process  activities  (including  pre-  and  post-AoA 
activities). 

(2)  Identify  issues  in  the  current  AoA  process. 

(3)  Identify  potential  solutions  to  issues,  as  well  as  the  organizations  with  the  authority  to 
enact  the  solutions. 

(4)  Lay  out  the  various  decision  makers  who  use  AoAs  and  supporting  analyses,  their  needs, 
and  the  potential  consequence  of  meeting  or  failing  to  meet  their  needs. 

THE  SCOPE  OF  THE  PROJECT  includes: 

(1)  "As  Is"  process  mapping.  What  are  the  inputs,  outputs,  timelines,  and  responsibilities 
for  each  step  of  the  process?  What  constraints  are  there  to  the  "As  Is"  process? 

(2)  "To  Be"  process  improvement  and  layout.  What  changes  are  needed?  Who  can  affect 
the  changes?  What  are  the  expected  time  saves/cost/risk  of  any  changes? 

(3)  Utilization  of  case  studies  for  both  “good”  and  “challenged”  AoAs  to  identify  lessons 
learned.  What  does  “right”  look  like?  What  problems  were  encountered  in  recent  AoAs?  How 
do  we  eliminate,  mitigate,  or  solve  these? 

(4)  Utilization  of  U.S.  Army  Training  and  Doctrine  Command  (TRADOC)  Analysis  Center 
(TRAC’s)  AoA  Primer  and  previous  acquisition  reviews  (Decker- Wagner,  etc.). 

(5)  Utilization  of  interviews. 

THE  PRINCIPAL  FINDINGS  are: 

(1)  The  AoA  is  burdened  by  "immature"  Initial  Capabilities  Documents  (ICD).  The  AoA 
must  then  refine  and  prioritize  capability  gaps,  requirements,  and  concepts  of  employment. 

(2)  Anny  leadership  does  not  inform  the  scope  of  the  AoA. 

(3)  For  developmental  systems,  the  Army  must  develop  data,  scenarios,  and  models  and 
simulations  (M&S)  to  support  analyses.  These  items  have  very  long  lead  times. 

(4)  Staff  friction,  sequential  reviews,  and  mission  creep  cause  delays. 
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THE  PRINCIPAL  RECOMMENDATIONS  are: 

(1)  TRADOC  ensures  a  Capabilities-Based  Assessment  (CBA)  is  conducted  to  standard  by 
those  that  developed  the  requirements  and  capabilities. 

(2)  Use  existing  processes  to  prioritize  gaps  (e.g.,  Capability  Portfolio  Review  (CPR)). 

(3)  The  Army  Requirements  Oversight  Council  (AROC)  reviewing  the  ICD  must  be  a 
decision-making  forum  authorized  to  scope  the  AoA  and  isolate  concepts  for  refinement. 

(4)  The  AROC  Memorandum  (AROCM)  must  codify  the  decisions  to  constrain  the  AoA’s 
scope. 

(5)  The  G-3  Capabilities  Integration  (Cl)  staff  (now  organized  under  G-8  FDS  (Force 
Development  -  Studies,  Analysis,  and  Technology  Division)  and  referred  to  in  this  document  as 
G-3  Cl,  Department  of  the  Anny,  Military  Operations  (DAMO)-CI,  or  Cl)  continues  to  negotiate 
early  with  Office  of  the  Secretary  of  Defense  (OSD)  Cost  Assessment  and  Program  Evaluation 
(CAPE)  to  ensure  scope  of  AoA  is  workable. 

(6)  AoA  guidance  should  clearly  identify  decisions  being  supported  and  reflect  AROC 
issues. 

(7)  Jump-start  AoAs  by  establishing  a  predictable  resourcing  strategy  and  AoA  forecast  to 
preclude  cold  start  of  M&S,  data,  and  scenarios. 

(8)  Continue  central  coordination  of  AoAs  that  are  planned  and  in  progress. 

(9)  Codify  decisions  (AROC  and  Study  Advisory  Group  (SAG))  to  minimize  mission  creep. 

(10)  Ensure  appropriate  SAG  Chair  and  Membership. 

(11)  Conduct  parallel  reviews  in  TRADOC,  Headquarters,  Department  of  the  Anny 
(HQDA),  and  OSD  CAPE. 

THE  PROJECT  EFFORT  was  conducted  by  Ms.  Renee  G.  Carlucci,  Center  for  Army 
Analysis,  Resource  Analysis  Division. 

COMMENTS  AND  QUESTIONS  may  be  sent  to  the  Director,  Center  for  Army  Analysis, 
ATTN:  CSCA-RA,  6001  Goethals  Road,  Suite  102,  Fort  Belvoir,  VA  22060-5230. 
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1  INTRODUCTION 

1.1  Analysis  of  Alternatives  Process  Review 


In  late  March  of  2016,  the  Anny  G-8  asked  the  Center  for  Army  Analysis  (CAA)  to  lead  an 
effort  to  help  the  Army  figure  out  how  to  shorten  the  Analysis  of  Alternatives  (AoA)  timeline. 
For  those  not  familiar  with  what  an  AoA  is,  an  AoA  is  an  analytical  comparison  of  the 
operational  effectiveness,  cost,  and  risks  of  proposed  materiel  solutions  to  gaps  in  operational 
capability.  An  AoA  documents  the  rationale  for  identifying  and  recommending  a  preferred 
solution  or  solutions  to  the  identified  shortfall.  Those  familiar  with  AoAs  in  the  Army  might 
ask,  “Why  was  CAA  asked  to  do  this  study?  They  have  nothing  to  do  with  AoAs.”  Exactly. 
Since  CAA  is  not  involved  at  all  in  the  AoA  process,  the  CAA  study  leads  could  examine  the 
AoA  process  activities  with  fresh  eyes  and  be  an  independent  arbiter  for  change 
recommendations. 

The  upfront  sections  in  this  report  cover  the  background  and  purpose  of  AoAs,  what  they 
typically  include,  and  how  the  Office  of  Cost  Assessment  and  Program  Evaluation  (CAPE) 
detennines  “sufficiency”  of  the  AoA  as  the  gate  for  milestone  (MS)  approval.  AoAs  are  required 
by  statute  for  Acquisition  Category  I  (ACAT  I)  programs.  There  are  also  a  good  number  of 
documents  required  by  statute  or  regulation  at  each  MS. 

A  2013  CAPE  study  examining  47  AoAs  from  2008  to  2013  from  all  Services  showed  that  AoA 
durations  ranged  from  3  months  to  3  years  with  an  average  of  1.5  years,  with  the  cost  ranging 
from  $1.3M  to  $15M  each  and  an  average  of  $5M.  We  don’t  have  the  Anny  figures  yet,  but  a 
2014  Air  Force  study  found  that  “AoAs  cost  approximately  one  half  of  one  tenth  of  one  percent 
of  total  program  costs  (e.g.,  ~$110M  in  AoA  support  impacted  ~$350B  in  decision-making).” 
The  average  AoA  duration  for  the  last  10  Army  AoAs  was  ~13  months,  with  another  2  to  4 
months  for  staffing  and  “sufficiency”  detennination  depending  on  whether  the  program  had 
Army  or  Office  of  the  Secretary  of  Defense  (OSD)  oversight. 

This  process  review  found  that  no  Army  AoA  has  ever  held  up  a  Defense  Acquisition  Board 
(DAB)  schedule.  AoA  perfonners  have  told  us  that  they  can  always  do  an  AoA  quicker,  but  that 
those  shortcuts  can  lead  to  a  system  that  is  unattainable  or  unaffordable.  Based  on  all  that  was 
learned  in  the  course  of  this  study,  the  challenges  and  recommendations  can  be  bundled  in  about 
five  areas  in  three  major  categories. 
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1.2  AoA  Process  Improvement  (PI)  Study  Advisory  Group  Membership 

•  Army  G-3,  Capabilities  Integration  (Cl) 

•  Army  G-8,  Force  Development  (FD) 

•  Army  G-8,  Program  Assessment  &  Evaluation  (PAE)  (FD,  PAE,  CAA) 

•  Army  G-8,  Center  for  Army  Analysis  (CAA) 

•  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics  and  Technology)  ASA(ALT) 

•  The  Office  of  Cost  Assessment  and  Program  Evaluation  (CAPE) 

•  TRADOC  Analysis  Center  (TRAC) 

•  Army  Materiel  Systems  Analysis  Activity  (AMSAA) 

•  Army  Research  Laboratory's  (ARL  )  Survivability/Lethality  Analysis  Directorate 
(SLAD)  ARL/SLAD 

•  The  Deputy  Assistant  Secretary  of  the  Army  for  Cost  and  Economics  (DASA-CE) 

•  Army  Capabilities  Integration  Center  (ARCIC) 

•  Army  Test  and  Evaluation  Command  (ATEC) 

•  Army  Evaluation  Center  (AEC) 

Figure  1.  AoA  PI  Study  Advisory  Group  Membership. 

For  this  study,  we  engaged  the  community  involved  in  the  AoA  process  from  all  perspectives, 
including  OSD  CAPE.  We  examined  the  guiding  documents,  the  numerous  previous  AoA 
reviews,  collected  case  study  information,  and  conducted  interviews. 

Figure  1  above  shows  the  major  organizations  that  were  involved  in  the  study. 
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1.3  Background 
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Figure  2.  Army  Programs  (2010  Army  Acquisition  Review). 

The  Army  has  had  a  poor  acquisition  track  record  with  too  many  cancellations,  schedule 
slippages,  cost  over-runs,  and  failures  to  deliver  timely  solutions  to  the  warfighters’ 
requirements.  This  has  led  to  numerous  acquisition  reviews,  National  Defense  Authorization  Act 
(NDAA)-directed  acquisition  refonn,  Congressional  scrutiny,  and  calls  from  Army  senior  leaders 
for  greater  control.  Figure  2  above  captured  in  the  “Army  Strong:  Equipped,  Trained  and 
Ready”,  Final  Report  of  the  2010  Army  Acquisition  Review  (otherwise  known  as  the 
Decker/Wagner  Report)  highlights  the  high  percentage  of  Anny  program  cancellation  and 
delays.  The  root  causes  for  troubled  and  terminated  programs  usually  stem  from  the 
developmental  planning  period  from  before  Materiel  Development  Decision  (MDD)  to  MS  A 
and  MSB. 


In  accordance  with  DoD  5000.02  (January  7,  2015)  and  the  Weapon  Systems  Acquisition 
Refonn  Act  (WSARA)  of  2009:  AoAs  are  a  statutory  requirement  for  Major  Defense 
Acquisition  Programs  (MDAPs),  Major  Automated  Information  Systems  (MAIS)  programs,  and 
all  Automated  Information  Systems  (AIS)  programs  at  Milestone  A.  Updates  are  required 
through  Milestone  C  (or  Milestone  B  if  there  is  no  Milestone  C)  for  MAIS  programs,  and  all  AIS 
programs. 

AoAs  consider  a  broad  set  of  solutions,  key  trades  among  cost,  schedule,  performance, 
affordability  analysis,  risk  analysis,  and  planning  for  risk  mitigation. 

The  AoA  will  focus  on  identification  and  analysis  of  the  proposed  alternatives;  measures  of 
effectiveness;  key  trades  between  cost  and  capability;  total  life-cycle  cost,  including  sustainment; 
schedule;  concepts  of  operations;  and  overall  risk.  The  AoA  will  inform  and  be  informed  by 
affordability  analysis,  cost  analysis,  sustainment  considerations,  early  systems  engineering 
analyses,  threat  projections,  and  market  research. 
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There  are  two  purposes  for  an  AoA:  1)  to  provide  infonnation  on  the  cost-effectiveness  of 
alternatives  considered  with  regard  to  mitigating  identified  capability  gaps  and  2)  to  inform 
development  of  requirements  documents  (primarily  the  Capability  Development  Document 
(CDD). 

The  primary  purpose  of  an  AoA  is  to  enable  the  Milestone  Decision  Authority  (MDA)  to  make 
an  infonned  acquisition  decision;  this  is  normally  the  Anny  Acquisition  Executive  (AAE)  or  the 
Defense  Acquisition  Executive  (DAE).  In  this  role,  the  AoA  is  a  decision  support  analysis 
evaluating  technologies  or  systems  to  provide  information  on  the  cost-effectiveness  of  the 
options  or  alternatives  considered  with  regard  to  mitigating  identified  capability  gaps. 

The  second  purpose  of  the  AoA  is  to  inform  development  of  requirements  documents  (primarily 
the  CDD).  In  this  role,  the  AoA  tests  the  veracity  of  draft  requirements  and  identifies  the 
system/technology  attributes  of  importance  and  the  recommended  values  of  those  attributes  to 
effectively  mitigate  the  identified  capability  gaps. 

1.4  Problem,  Purpose,  and  Objectives 


•  Problem  Statement 

•  The  Army  requires  AoAs  and  supporting  analyses  that  satisfy  decision-makers'  critical 
information  requirements  that  are  delivered  in  a  timely  manner. 

•  Purpose 

•  Develop  an  approach  to  focus  AoA  activities  to  most  efficiently  and  effectively  deliver 
results  that  meet  decision-maker  needs. 

•  Objectives 

•  Lay  out  all  the  analysis-related  AoA  process  activities  (including  pre-  and  post-AoA 
activities). 

•  Identify  issues  in  the  current  AoA  process. 

•  Identify  potential  solutions  to  issues,  as  well  as  the  organizations  with  the  authority  to 
enact  the  solutions. 

•  Lay  out  the  various  decision  makers  who  use  AoAs  and  supporting  analyses,  their 
needs,  and  the  potential  consequence  of  meeting  or  failing  to  meet  their  needs. 

Figure  3.  Problem,  Purpose,  and  Objectives. 

Some  Army  senior  leaders  have  complained  that  AoAs  take  too  long  and  cost  too  much.  Lt.Gen. 
John  M.  Murray,  Deputy  Chief  of  Staff,  G-8,  requested  that  CAA  take  on  this  effort  to  help  the 
Anny  figure  out  how  to  shorten  the  AoA  timeline.  The  problem  statement  was  later  refined  to 
the  one  shown  in  Figure  3  above. 

Merely  shortening  the  timeline  of  the  AoA  is  not  sufficient.  Many  of  the  AoA  perfonners  agreed 
that  they  could  always  take  less  time  to  complete  an  AoA.  However,  that  would  often  result  in 
shortcuts  being  taken  that  can  affect  the  results  of  the  AoA.  CAPE  leaders  interviewed  reported 
that  when  they  don’t  get  a  good  rigorous  AoA,  the  perfonners  often  end  up  having  to  redo  it. 
They  also  reported  that  AoAs  are  often  started  with  insufficient  work  on  the  requirements  and 
that  this  is  a  problem  they  are  seeing  from  all  the  Services.  Shortcuts  taken  in  the  AoA  or 
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insufficient  work  done  in  the  requirements  process  has  often  lead  to  a  system  that  is  unattainable 
or  unaffordable. 

So  the  question  asked  became,  “How  can  the  Army  improve  its  process  for  conducting  AoAs  to 
ensure  that  DoD  has  appropriately-scoped,  sufficient,  and  timely  analysis  to  infonn  requirements 
development  and  acquisition  decisions  for  the  highest  priority  Army  systems,  while  maximizing 
efficiency  of  scarce  analytic  resources?”  We  also  learned  that  there  are  now  many  stakeholders 
and  AoA  customers  that  actively  use  the  analysis  (e.g.,  Acquisition  Executives,  Joint 
Requirements  Oversight  Council  (JROC),  Planning,  Programming,  Budgeting,  and  Execution 
(PPBE)  process,  and  Congress).  All  of  these  are  or  have  decision  makers  that  use  AoAs  and  can 
significantly  affect  a  program. 

1.5  Scope/Methodology 


•  Week  1- 25  March  to  1  April  2016 

•  Identification  of  Study  Advisory  Group  (SAG)  Participants/  Division  of  Labor 

•  Case  Study  Identification 

•  CAA  reviews  Decker-Wagner  recommendations 

•  Confirm  SAG  Battle  Rhythm 

•  Weekly  VTCs,  Telecoms,  Virtual 

•  Identify  alternate  DTGs 

•  Plan  Interviews 

•  Weeks  2  &3-4to  15  April  2016 

•  Conduct  Case  Study  Analysis 

•  Continue  Literature  Review 

•  Conduct  Planned  Interviews 

•  Map  Process  Activities 

•  Conduct  SAG  Findings  Review  to  Establish  Consensus 

•  Week  4 -18  to  22  April  2016 

•  Draft  &  Staff  Briefing  of  Results 

•  Draft  Report 

Figure  4.  Scope/Methodology. 

Given  the  short  duration  of  the  study,  just  30  days,  the  study  team  chose  to  utilize  a  Study 
Advisory  Group  (SAG)  composed  of  all  the  pertinent  stakeholders.  This  report  encompasses 
information  provided  by  members  across  the  SAG.  Our  methodology,  shown  in  Figure  4  above, 
was  fairly  simple.  We  utilized  the  advice  of  the  SAG  to  identify  a  good  mix  of  AoA  case  studies 
(encompassing  all  ACAT  levels  and  those  considered  good  examples  and  those  that 
encompassed  challenges),  reviewed  guiding  literature  and  previous  AoA  reviews,  conducted 
interviews  with  stakeholders,  and  mapped  the  process  activities  and  durations. 

We  tasked  the  SAG  members  with  providing  the  information  required  on  case  studies  and 
process  activities  with  timelines  and  conducted  weekly  SAGs  to  review  ongoing  findings  and 
establish  consensus.  The  SAG  members  included  AoA  performers  (U.S.  Anny  Training  and 
Doctrine  Command  (TRADOC)  Analysis  Center  (TRAC),  Army  Materiel  Systems  Analysis 
Activity  (AMSAA),  Anny  Capabilities  Integration  Center  (ARCIC),  Anny  Research 
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Laboratory’s  Survivability/Lethality  Analysis  Directorate  (ARL/SLAD)),  the  Deputy  Secretary 
of  the  Army  for  Cost  and  Economics  (DASA-CE),  OSD  CAPE,  Assistant  Secretary  of  the  Anny 
(Acquisition,  Logistics  and  Technology)  ASA(ALT),  G-3  Capabilities  Integration  (Cl),  and  G-8. 

The  SAG  agreed  that  the  study  should  address  AoA  prioritization  and  scope.  Whoever  sets  the 
priorities  must  coordinate  with  G-3,  G-8,  and  ASA(ALT).  Whoever  governs  the  scope  of  the 
AoA  should  be  able  to  coordinate  with  CAPE,  G-8,  G-3,  ARCIC,  TRAC,  and  AMSAA. 

The  study  team  should  examine  the  "As  Is"  process  mapping.  What  are  the  inputs,  outputs, 
timelines,  and  responsibilities  for  each  step  of  the  process?  What  constraints  are  there  to  the  "As 
Is"  process?  In  order  to  detennine  the  "To  Be"  process  improvement  and  layout:  What  changes 
are  needed?  Who  can  affect  the  changes?  What  are  the  expected  time  saves/cost/risk  of  any 
changes?  Regarding  utilization  of  case  studies  for  both  “good”  and  “challenged”  AoAs,  What 
does  “right”  look  like?  What  problems  lead  to  a  “challenged”  AoA?  How  do  we  eliminate, 
mitigate,  or  solve  these?  The  study  team  should  utilize  TRAC’s  AoA  Primer  and  previous 
acquisition  reviews  (Decker- Wagner,  etc.). 

The  study  team  should  also  examine  development  of  the  AoA  Study  Directive  and  Guidance. 
What  needs  to  be  included?  How  should  it  be  developed?  It  should  cover  inclusion  of 
statutory/regulatory  requirements  and  elicited  decision  maker  analytic  needs. 

Regarding  the  technical  review  of  a  given  AoA  including  the  AoA’s  SAG:  What  should  the 
reviews  be  geared  to  accomplish?  Are  the  requirements  and  gaps  defined  and  scoped  sufficiently 
prior  to  the  start  of  the  AoA  process?  Is  the  AoA  covering  the  right  alternatives,  requirements, 
mission  profiles,  and  scenarios? 

The  study  must  look  at  improving  the  SAG  process  with  the  first  SAG  used  to  narrow  down  the 
requirements.  There  is  a  need  for  up-front  mission  analysis  and  up-front  Senior  Leader  guidance 
from  both  Anny  leadership  as  well  as  OSD  leadership  -  before  proceeding  on  the  AoA.  The 
number  of  alternatives  and  additional  tasks  added  throughout  AoA  process  drive  the  scope  and 
timeline.  Does  the  AoA  sufficiently  address  the  research  questions  to  infonn  the  next  acquisition 
decision?  Who  should  participate?  When  should  reviews  be  conducted  (iterative)?  What  should 
be  the  relationship  between  the  Army  Requirements  Oversight  Council  (AROC)  and  the  SAG? 

Funding  for  AoAs  and  pre-MDD  analytical  activities  has  long  been  a  problem.  This  study 
should  identify  measures  to  improve  the  funding  process.  What  are  the  options  for  whom/what 
office  should  synchronize  AoAs  for  the  Anny?  Should  we  have  a  single  office  responsible  for 
the  prioritization,  scope/measurement  space,  and  study  directive/guidance  for  AoA  activities? 
Finally,  this  effort  needs  to  consider  pre/supporting  AoA  activities,  i.e.,  analytic  efforts  to 
develop  the  Initial  Capabilities  Documents  (ICD),  CDD,  and  Capability  Production  Document 
(CPD). 


6  •  INTRODUCTION 


UNCLASSIFIED 


AoA  PI 


UNCLASSIFIED 


CAA-2016058 


2  AoA  PRIMER 

2.1  The  Capabilities-Based  Assessment  to  AoA  Linkage 


The  Joint  Capabilities  Integration  and  Development  System  (JCIDS)  Manual  describes  the 
Capabilities-Based  Assessment  (CBA)  as  the  analytic  basis  to  identify  capability  requirements 
and  associated  capability  gaps  prior  to  development  and  submission  of  capability  requirement 
documents  for  review  and  validation.  The  CBA  is  the  start  point  for  the  deliberate  requirements 
process  and  leads  to  the  development  of  the  ICD.  This  analysis  is  meant  to  define  the  mission, 
identify  capabilities  required,  determine  the  attributes/standards  of  the  capabilities,  identify  gaps, 
assess  operational  risk  associated  with  the  gaps,  prioritize  the  gaps,  identify  and  assess  potential 
non-materiel  solutions,  and  provide  recommendations  for  addressing  the  gaps.  The  intent  of  a 
CBA  may  also  be  satisfied  through  one  or  more  other  studies  or  analyses,  as  long  as  the 
analytical  rigor  and  breadth  of  analysis  is  covered  by  the  collective  analytical  efforts. 

The  CBA  does  not  provide  specific  recommendations  as  to  a  particular  materiel  solution,  but 
rather  provides  a  more  general  recommendation  as  to  the  type  of  materiel  solution  (whether  it  is 
an  incremental  improvement  to  an  existing  capability,  or  an  entirely  new  capability).  In  this 
way,  the  ICD  is  used  to  establish  boundary  conditions  for  the  scope  of  alternatives  to  be 
considered  in  the  subsequent  AoA. 

TRADOC,  with  many  stakeholders,  conducts  a  centralized  Capabilities  Needs  Analysis  (CNA), 
which  assesses  the  needs  of  the  whole  Anny,  not  specific  systems.  They  don’t  have  the 
resources  to  do  systems-level  requirements  analysis  for  all  systems.  However,  the  TRADOC 
Centers  of  Excellence  (CoEs)  conduct  some  special  CBAs  via  functional  analyses  to  prioritize 
gaps  and  solutions.  These  are  narrower  in  scope  and  for  specific  gaps.  These  are  integrated  with 
the  overarching  CNA.  Other  analyses  and  activities,  such  as  the  Campaign  of  Learning  (CoL) 
and  the  Anny  Warfighting  Assessment  (AW A)  are  also  inputs  to  the  CNA.  The  CNA  is 
TRADOC’ s  primary  future  force  prioritization  tool  and  serves  as  TRADOC ’s  position  in 
informing  the  Program  Objective  Memorandum  (POM)  (primarily  the  Equipping  (EE), 
Sustaining  (SS)  and  Training  (TT)  Program  Evaluation  Groups  (PEGs)  and  Long-range 
Investment  Requirements  Analysis  (LIRA).  It  serves  as  the  Anny’s  integrated  CBA. 

AoAs  require  a  great  deal  of  information  on  the  proposed  alternatives  such  as  organizational  and 
operational  concepts,  employment  tactics,  techniques,  and  procedures  (TTP),  market  research  on 
technologies,  and  analytic  tool  development  (to  represent  the  new  concepts  and  TTP).  This  is  all 
work  that,  if  not  done  in  the  proponents  CBA  or  the  Program  Executive  Office  (PEO)  or 
Program  Manager  (PM)  market  research  for  technology  solutions,  will  add  a  significant  amount 
of  time  on  the  front  end  of  any  AoA. 
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2.2  Typical  AoA 


•  cost  effectiveness  of  the  technologies  or  systems  under  consideration. 

•  cost-schedule-performance  trades  analysis  (introduced  by  WSARA  2009). 

•  performance  level  analysis  (specific  to  the  technologies  under  consideration). 

•  operational  benefit  analysis  to  determine  to  what  extent  the  technology/system 
mitigates  the  capability  gap(s)  identified  in  the  ICD. 

•  lifecycle  cost  analysis  for  the  expected  life  of  the  technology/system  (using  the  cost 
analysis  rules  from  OSD  Cost  Analysis  and  Program  Evaluation  (DCAPE)). 

•  risk  assessments  on  the  technologies/systems  in  the  areas  of  cost,  performance, 
schedule,  and  operational  benefit/effectiveness. 

•  analysis  of  technology  readiness. 

•  reliability  and  sustainment  analysis. 

•  Assess  Fully  burdened  cost  of  fuel  and  Operational  Energy  analysis  (if  applicable). 

•  Affordability  Analysis,  in  the  Army  this  is  conducted  in  the  Army  G-8. 


Where  relevant,  OSD  CAPE  requires  analysis  be: 

•  Scenario-based,  preferably  in  a  region  of  key  interest. 

•  Utilize  quantitative  analysis  over  qualitative  analysis. 

Figure  5.  Typical  AoA. 


A  typical  AoA  includes  the  elements  shown  in  Figure  5  above.  As  discussed  above,  without  the 
knowledge  necessary  to  conduct  analysis  on  the  proposed  alternatives,  these  study  components 
will  be  difficult  or  impossible  to  develop.  There  are  no  rules  specifying  when  to  update  an  AoA. 
It  has  become  standard  OSD  CAPE  practice  to  develop  AoA  guidance  for  the  initial  MS  A  AoA 
that  is  very  broad  and  covers  the  life  of  the  acquisition  process.  The  intent  is  that  only  one  AoA 
is  needed,  however  this  has  led  to  problems  by  significantly  increasing  the  scope  of  the  analysis 
beyond  what  is  needed  for  the  milestone  decision  and  beyond  the  level  of  knowledge  developed 
and  available  at  the  time.  As  a  result,  AoA  updates  are  very  commonplace.  Some  reasons  for 
updates  are:  (1)  Knowledge  is  now  developed  and  available  to  address  decision  maker  questions 
that  could  not  be  addressed  in  an  earlier  AoA.  (2)  Threat  projections  have  changed  significantly 
since  the  last  AoA.  (3)  United  States  (U.S.)  force  concepts  and  operations  have  changed 
significantly  since  the  last  AoA.  (4)  Technology  (threat  or  U.S.)  has  significantly  changed  since 
the  last  AoA. 
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2.3  How  an  AoA  is  deemed  “sufficient” 


In  accordance  with  the  DoDI  5000.02  (page  126,  Enclosure  9),  DCAPE  provides 
a  memorandum  to  the  MDA  assessing*: 

1.  The  extent  to  which  the  AoA: 

a)  Examines  sufficient  feasible  alternatives. 

b)  Considers  tradeoffs  among  cost,  schedule,  sustainment,  and  required 
capabilities  for  each  alternative  considered. 

c)  Achieves  the  affordability  goals  established  at  the  MDD  and  with  what  risks. 

d)  Uses  sound  methodology. 

e)  Discusses  key  assumptions  and  variables  and  sensitivity  to  changes  in  these. 

f)  Bases  conclusions  or  recommendations,  if  any,  on  the  results  of  the  analysis. 

g)  Considers  the  fully  burdened  cost  of  energy  (FBCE),  where  FBCE  is  a 
discriminator  among  alternatives. 

2.  Whether  additional  analysis  is  required. 

3.  How  the  AoA  results  will  be  used  to  influence  the  direction  of  the  program. 

‘from  DoDI  5000.02,  Enclosure  9,  page  126. 


Figure  6.  How  an  AoA  is  deemed  “sufficient”. 

From  some  of  the  guiding  literature,  Department  of  Defense  Instruction  (DoDI)  5000.02,  the 
AoA  assesses  potential  materiel  solutions  that  could  satisfy  validated  capability  requirement(s) 
documented  in  the  Initial  Capabilities  Document,  and  supports  a  decision  on  the  most  cost 
effective  solution  to  meeting  the  validated  capability  requirement(s).  In  developing  feasible 
alternatives,  the  AoA  will  identify  a  wide  range  of  solutions  that  have  a  reasonable  likelihood  of 
providing  the  needed  capability.  Figure  6  above  describes  the  criteria  for  CAPE  to  assess 
“sufficiency”  of  an  AoA. 

While  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  3 170.011  is  silent  on  the  purpose 
of  the  AoA,  the  below  passage  gives  an  idea  of  the  AoA  purpose:  "The  validated  ICD  is  a  critical 
entry  criterion  for  the  MDD,  and  guides  the  Sponsor  [Materiel  Solution  Analysis  (MSA)]  phase 
activities  and  assessment  of  potential  materiel  solutions  through  an  AoA  or  other  studies. . . .  (b) 
Post-AoA  Review.  Following  Sponsor  completion  of  the  AoA,  the  post-AoA  review  provides 
the  validation  authority  and  other  stakeholders  the  opportunity  to  assess  how  the  different 
alternatives  address  the  validated  capability  requirements  and  associated  capability  gaps,  and  at 
what  life  cycle  costs.  The  post-AoA  review  shall  be  completed  in  sufficient  time  to  pennit 
Sponsor  preparation  of  a  draft  CDD  or  similar  documentation  prior  to  MS  A...  The  post-AoA 
review  is  not  a  validation  of  the  AoA  results,  but  rather  infonns  the  validation  authority's  advice 
to  the  [MDA]  on  the  AoA  results,  recommended  altemative(s),  and  proposed  [Key  Perfonnance 
Parameters  (KPPs)],  Key  System  Attributes  (KSAs),  and  Additional  Performance  Attributes 
(APAs)." 
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In  short  the  CAPE  assessment  evaluates  if  the  study  team  did  what  was  specified  in  the  guidance; 
how  well  the  work  was  done;  and  if  the  work  provides  relevant,  credible  analysis  to  support  the 
milestone  decision. 

One  important  note  is  that  the  AoA  does  not  make  the  acquisition  decision  but  rather  provides 
information  on  the  cost-effectiveness  of  the  alternatives  for  the  decision  makers  to  combine  with 
other  relevant  information  to  make  an  infonned  acquisition  decision. 

2.4  Milestone  A  Documents  Required 


Example  from  existing  program 


Document 

Status 

(Color 

Code) 

Owner 

Current 

Location 

Final 

Approval 

Authority 

If  appl, 
PEO 

Approved? 

(Yes/No/N/A) 

If  No, 
Expected 
Date  PEO 
Approval 

Expected 

Date, 

Approval 

Authority 

Required 

Date, 

Approval 

Authority 

Comments  and  Mitigation 

2366  Certification  Memorandum 
(S) 

PM/USD  ATL 

PM 

MDA 

N/A 

N/A 

15  Apr  16 

30  Jun  16 

Draft  due  prior  to  OIPT 

-BCA 

PM/USD  ATL 

PM 

MDA 

N/A 

N/A 

10  Apr  16 

15  Apr  16 

Draft  due  prior  to  OIPT 

Acquisition  Decision 

Memorandum  (R) 

PM/USD  ATL 

PM 

MDA 

No 

22  April 

30  May  16 

30  May  16 

Draft  due  prior  to  OIPT,  DRM  will 
jrovide  formal  input  16  Jun  16 

Acquisition  Strategy  (S) 

PM 

PM 

MDA 

No 

18  Mar  16 

15  May  16 

1  Jun  16 

In  formal  staffing  as  of  15  Mar  16 

Affordability  Analysis  (R) 

■ 

G8  P&E 

G8  P&E 

MDA 

N/A 

N/A 

19  May  16 

19  May16 

Goal  established  at  MDD;  validated 
during  CRB  process 

Analysis  of  Alternatives  (S) 

TRADOC 

DCAPE 

DCAPE 

N/A 

N/A 

31  Mar  16 

31  Mar  16 

Signed  by  TRADOC  on  17  Feb  16; 
pending  Sufficiency  Memo 

Capability  Development 

Document  (R) 

DA  G3/5/7 

JROC  Approved  PAC-3  CDD  (15  Aug 
13),  Need  user  input 

Clinger-Cohen  Act  Compliance 
(S) 

■ 

PM 

PM 

CIO 

No 

12  Apr  16 

1  Jun  16 

15  Jun  16 

Finalized  after  Cyber  Security  Strategy 
approval 

Concept  of 

Operations/Operational  Mode 
Summary/  Mission  Profile 
(CONOPS/OMS/MP)  (Update)  (R) 

TCM 

TCM 

DA  G3/5/7 

N/A 

N/A 

18  Mar  16 

15  Jun  16 

Pending  TCM  updates 

Contractor  Cost  and  Software 

Data  Reporting 

PM 

PM 

DCARC 

Not  required  for  TMRR. 

Core  Logistics 

Determination/Core  Logistics  & 
Sustaining  Workloads  Estimate 
(S) 

■ 

PM 

APEO  LOG 

MDA 

No 

18  Mar 

1  Jun  16 

15  Jun  16 

Pending  AMCOM  brief  25  April  16,  prior 
to  signature.  Only  CLD  required  for  MS 

A;  in  the  LCSP 

Cost  Analysis  Requirements 
Description  (CARD)  (R) 

PM 

ASA(ALT) 

ASA(ALT) 

Yes 

26  Feb  16 

18  May  16 

1  Jun  16 

In  formal  staffing.  Addendum  due 

24  Mar  16  to  PEO 

Cybersecurity  Strategy  (S) 

PM 

APEO-  OPNS 

CIO 

No 

2  Apr  16 

1  May  16 

27  May  16 

Appendix  to  the  PPP 

DoD  Component  Cost  Estimate 
(Project  Office  Estimate)  (R) 

PM 

PM 

ASA(FMC) 

N/A 

N/A 

5  Apr  16 

_ 

18  Apr  1( 

Complete 

DoD  Component 

Cost  Position  (Army  Cost 

Position)  (R) 

DASA-CE 

DASA-CE 

DASA-CE 

N/A 

N/A 

18  May  16 

25  May  1 

Resolvable  Schedule  Issues 

Cannot  Meet  Schedule 

Figure  7.  Milestone  A  Documents  Required  (1  of  2). 
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Document 

Status 

(Color 

Code) 

Owner 

Current 

Location 

Final 

Approval 

Authority 

If  appl, 
PEO 

Approved? 

(Yes/No/N/A) 

If  No, 
Expected 
Date  PEO 
Approval 

Expected 

Date, 

Approval 

Authority 

Required 

Date, 

Approval 

Authority 

Comments  and  Mitigation 

Frequency  Allocation  Submission 
(DD  Form  1494)  (S) 

PM 

Stage  1  submitted  12  March  2015  to 

ASMO 

Full  Funding  Certification 
Memorandum  (R) 

G8  P&E 

PM 

AAE/ 

ASA(FMC) 

No 

18  May 

23  May  16 

31  May  16 

Must  be  signed  by  AAE  and  ASA(FMC); 
follow  cost  position  approval 

Independent  Cost  Estimate  (S) 

CAPE 

28  Jun  16 

30  Jun  16 

Prepared  by  DCAPE  CAIG 

Item  Unique  Identification  (R) 

PM 

Part  of  SEP  and  Annex  to  LCSP. 

Per  APEO  LOG  not  required  for  MS  A 

Life-Cycle  Mission  Data  Plan  (R) 

PM 

DIA 

DIA 

N/A 

N/A 

15  Apr  16 

15  Apr  16 

DIA  IMDC  stated  they  will  not  hinder 

MS  decision;  will  provide  conditional 
acceptance  if  required 

Life-Cycle  Sustainment  Plan  (R) 

PM 

PEO  DASM 

MDA 

No 

23  Mar  16 

1  Jun16 

15  Jun  16 

Pending  AMCOM  brief  25  April  16,  prior 
to  signature. 

Program  Protection  Plan  (R) 

PM 

PM 

MDA 

No 

31  Mar  16 

1  June 

15  Jun  16 

Pending  final  correction  from  Jerry 
Mangino 

Replaced  System  Sustainment 

Plan  (S) 

PM 

Not  required  per  APEO  LOG 

DRAFT-Request  for  Proposal  (R) 

PM/  ACC-NJ 

PM 

MDA 

No 

9  May  16 

30  Jun  16 

30  Jun  16 

Pending  AS  approval;  draft  will  be 
released  ~May;  RFP  after  DAB 

Should  Cost  Target  (R) 

PM 

PM 

MDA 

No 

29  Apr  16 

2  May  16 

16  May  16 

Predecessor:  CARD,  POE,  CCE,  ICE,  ACP 

Spectrum  Supportability  Risk 
Assessment  (R) 

PM 

PM 

CIO/G-6 

Yes 

7  Jan  16 

1  Apr  16 

15  Apr  16 

ASMO/CIO  G-6  will  provide  a  positive 
spectrum  supportability  assessment. 

System  Threat  Assessment 

Report  (STAR)  (R) 

PM 

Using  AAMD  STAR  validated  by  DIA  11 
March  2015 

System  Engineering  Plan  (SEP) 

(R) 

PM 

PEOCE 

DASD(SE) 

No 

23  Mar 

15  May  16 

31  May  16 

Document  was  submitted  to  DASM 

Technology  Targeting  Risk 
Assessment  (TTRA)  (R) 

- 

Complete 

Test  and  Evaluation  Master  Plan 
(TEMP)  ( R) 

PM 

PM 

DDTE/DOTE 

No 

2  May 

1  Jun  15 

On  Schedule 

_ 

Poenluahlp  RrhoHnlo  Icciioc 

1 

z 

Cannot  Meet  Schedule 

n 

Figure  8.  Milestone  A  Documents  Required  (2  of  2). 

Figure  7  and  Figure  8  above  show  all  the  documents  required  for  a  recent  program  at  Milestone 
A.  As  one  can  see,  there  are  many  products  required  prior  to  Milestone  A,  not  just  the  AoA. 
From  our  interviews  and  case  study  analysis,  we  could  find  no  example  of  an  AoA  delaying  a 
DAB. 
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2.5  Current  AoA  Landscape 


•  Ongoing  AoAs 

•  Army-led: 

•  1.  Lower  Tier  and  Missile  Defense  Capability  (LTAMDC)  (ACAT 1,  CAPE  oversight)  [Final 
Report  complete;  the  Army  considers  the  AoA  complete  once  CAPE  assesses  analytic 
sufficiency] 

•  2.  EAB  M113  Family  of  Vehicle  Replacement  (Congressionally-directed,  no  MDD  -  projected 
ACAT  1,  HQDA  oversight) 

•  3.  Dominating  Terrain  through  Terrain  Shaping  and  Engagement  (DMTTS&E),  Inc  1  (GATOR 
replacement)  (ACAT  ID,  CAPE  oversight) 

•  4.  Ground  Mobility  Vehicle  (GMV)  (ACAT  III,  TRADOC  Oversight) 

•  Joint  (Army-supporting):  5.  Cyber  UNIFIED  PLATFORM  (ACAT  I,  CAPE  oversight,  USAF-led) 

•  Pre-MDD  work  for  AoAs  not  yet  started: 

•  1.  Next  Generation  Biometric  Collection  Capability  (NXGBCC)  (ACAT  1C,  HQDA  oversight)  [AoA 
Study  Guidance  staffing  complete  -  signature  imminent] 

•  2.  Future  of  Vertical  Lift,  Medium  (ACAT  ID,  CAPE  oversight)  [AoA  Study  Guidance  in 
Development] 

•  3.  Automated  Convoy  Operations  (ACAT  ID,  CAPE  oversight) 

•  4.  Electronic  Warfare  -  Defensive  Electronic  Attack  (EW-DEA)  (ACAT  1,  CAPE  oversight) 

•  5.  Common  Remotely  Operated  Weapon  Station  (CROWS)  Inc  2  (ACAT  1,  CAPE  oversight) 

•  6.  Global  Force  Information  Management  (GFIM)  (ACAT  III,  HQDA  Oversight) 

•  7.  Mobile  Protected  Firepower  (MPF) 


Figure  9.  Current  AoA  Landscape. 

At  the  time  of  our  AoA  Process  Improvement  study,  G-3  Cl  advised  us  that  the  current  AoA 
landscape  consisted  of  the  ongoing  AoAs  listed  in  Figure  9  above  as  well  as  Pre-MDD  work  for 
future  AoAs  also  listed  above.  There  is  also  a  much  longer  list  maintained  of  potential  AoAs  for 
Fiscal  Year  (FY)  2018  and  FY  2019  (not  shown). 
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3  INTERVIEWS 

3.1  Overview 


•  Mr.  Peter  Bechtel,  COL  George  Seaward,  Ms.  Julia  Sharkey  (G-3  Cl) 

•  Mr.  Edward  Koucheravy  (CAPE,  has  overseen  AoAs  from  most  of  the  Services) 

•  Dr.  Scott  Comes  (oversees  all  AoAs  with  CAPE  participation) 

•  MG  John  Ferrari  (G-8  PAE) 

•  COL  James  Schirmer  (Program  Manager) 

•  Ms.  Pamela  Blechinger,  Ms.  Bonnie  Mcllrath  (TRAC) 

•  MG  Neil  Thurgood  (ASA(ALT)) 

•  Mr.  James  Amato,  Mr.  David  Payne,  Ms.  Lindsay  Bear  (AMSAA) 

•  Mr.  Greg  Mannix  (SLAD) 

Figure  10.  Interviews. 

Due  to  the  short  timelines  of  the  study,  we  were  not  able  to  interview  all  of  the  desired 
stakeholders.  Figure  10  above  shows  the  list  of  individuals  that  we  formally  interviewed. 
Obviously,  all  participants  in  our  SAG  were  able  to  provide  regular  feedback  and  comments 
through  the  course  of  all  our  SAG  discussions  as  well.  Many  of  the  comments  received  during 
the  interviews  echo  those  things  highlighted  in  the  lessons  learned  and  other  AoA  reviews  we 
examined.  We  highlight  the  following  points  or  recommendations  heard  during  the  interviews 
aggregated  based  on  their  nature  and  generally  not  attributed. 

3.2  Pre-AoA  Activities 


Several  interviewees,  citing  specific  examples,  stressed  the  need  for  the  Army  to  do  more 
analysis  during  the  requirements  process.  Typically,  the  AoA  takes  the  requirements  as  semi¬ 
gospel  and  determines  the  best  way  to  meet  them.  The  Anny  needs  to  do  a  better  job  of  cost- 
informed  trades  before  it  launches  a  program.  There  was  a  recommendation  for  a  single  PM-like 
organization  with  engineers,  science  and  technology  expertise,  and  cost  analysts  to  oversee  work 
done  before  a  program  is  launched.  CAPE  has  formulated  a  series  of  recommendations  to 
improve  the  requirements  process  in  all  the  Services  and  using  a  CBA  with  increased  analytical 
rigor,  considering  how  affordability  affects  trade  space,  and  prioritizing  capability  gaps  based  on 
mission  consequences  to  help  determine  material  solution  trade  space.  They  also  recommend 
standing  up  a  Service-led  SAG  with  external  stakeholders  for  each  CBA,  similar  to  what  is  done 
for  the  AoA  (see  section  E-8). 

Another  concern  highlighted  is  the  lack  of  a  senior  analyst  for  the  Anny  to  assist  with  the 
oversight  and  prioritization  of  pre-AoA  analytical  activities.  At  one  time  the  Anny  had  a  Deputy 
Under  Secretary  for  Operations  Research  (DUSA-OR),  Mr.  Walt  Hollis,  to  fulfill  this  role.  He 
worked  to  improve  the  relationship  between  the  testers,  the  acquisition  community,  and  the 
analysts  as  well.  Ms.  Blechinger  is  currently  the  lead  analyst  for  TRADOC,  but  doesn’t  have 
any  oversight  role  in  ARCIC.  ARCIC’s  last  senior  executive  analyst  billet  held  by  Mr.  A1 
Resnick,  was  abolished  after  his  retirement.  Consequently,  the  most  senior  analyst  in  ARCIC  is 
at  the  Colonel  or  General  Schedule  (GS)-15  level.  A  recommendation  of  the  Decker-Wagner 
Study  was  to  restore  the  DUSA-OR  function. 
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3.3  AoA  Prioritization  and  Funding 


One  of  the  benefits  of  the  Capability  Portfolio  Review  (CPR)  process,  over  and  above  the 
AROC,  was  the  prioritization  of  gaps.  The  CNAs  are  getting  better  and  including  gaps  from 
Combatant  Command  Integrated  Priority  Lists  (IPLs)  and  from  Army  Commands  (ACOMs). 

The  portfolio  aspect  of  the  CPRs  allowed  senior  leaders  the  ability  to  emphasize  gaps.  That, 
along  with  a  requirements  to  resources  process,  allows  backwards  planning  and  programming  to 
resource  forthcoming  AoAs.  Unfortunately,  the  funding  plan  has  not  always  been  in  line  with 
senior  leader  guidance. 

When  ARCIC  was  created,  resources  were  pulled  out  of  TRADOC.  The  question  has  been  how 
much  workload  can  they  push  to  MDD  analysis?  A  better  assessment  of  their  capability  is 
needed.  The  acquisition  lane  has  a  role  in  this  as  well.  What  can  they  do  to  help?  There  is  a 
relationship  of  Science  and  Technology  (S&T)  guidance  to  pre-MDD  analysis  and  then  to  the 
AoAs.  How  can  we  connect  all  of  the  various  activities  (e.g.,  work  in  the  areas  of  Counter- 
Unmanned  Aerial  Surveillance  (UAS),  directed  energy,  lasers,  precision  fires,  known  threat 
vectors,  and  technology  demonstration  activities)  to  a  discrete  capability  that  a  future  PM  will 
use?  Forward-looking  guidance  needs  to  be  given  for  all  of  these  activities  and  the  activities 
need  to  be  federated  to  support  requirements.  While  G-3/5/7  currently  lays  out  a  prioritization 
for  AoAs,  these  are  not  always  in  line  with  senior  leader  priorities.  The  pre-AoA  work  being 
done  may  also  not  align  with  these  priorities.  Finally,  DoD  priority  targets  are  not  necessarily  in 
line  with  Anny  priority  targets.  Many  of  the  interviewees  cited  the  need  to  get  senior  leader 
priorities  and  questions  early  on. 

The  Anny  assumption  is  the  need  for  approximately  $10M  in  funding  per  year  for  AoAs. 

(TRAC  pointed  out  that  some  years  that  can  be  as  high  as  $20M,  which  they  have  accomplished 
using  mission  funding.)  Delays  in  distributing  AoA  funding  has  not  only  led  to  delays  in 
accomplishing  AoAs,  but  has  resulted  in  under-execution  and  marks  taken.  Several  interviewees 
cited  the  need  for  dedicated  funding  for  AoAs,  utilizing  a  1-  or  2-star  forum  for  periodic  reviews 
of  the  spending  plan  in  order  to  shift  funding  as  needed  between  AoA  perfonners.  Both  TRAC 
and  AMSAA  have  executed  their  ever-increasing  AoA  mission  with  a  combination  of  mission 
and  reimbursable  funds.  Providing  the  executing  agencies  with  funds  at  the  beginning  of  the  FY 
and  allowing  flexibility  between  AoAs,  depending  on  prioritization,  is  critical.  Ensuring  that 
procedures  are  in  place  to  properly  allocate  funds  to  the  agencies  making  up  the  team  is  also 
critical.  Recommend  TRAC  and  AMSAA  work  the  latter. 

3.4  AoA  Purpose 


The  AoA  is  more  than  just  the  acquisition  choice  of  A,  B,  or  C.  It  informs  understanding  of 
concept  of  operations  (CONOPS)  as  well  as  how  a  product  will  be  developed.  Here  the  choices 
and  data  analysis  can  infonn  the  trade  space.  Are  we  trying  to  prove  an  already  demonstrated 
technology  or  trying  to  understand  what  would  be  best?  For  example,  some  in  OSD  want  a  real 
long-range  weapon  -  but  it  probably  won’t  have  a  big  warhead.  The  AoA  really  informed  that 
debate.  The  second  purpose  of  the  AoA  is  to  build  consensus  within  the  Army  and  outside 
parties. 
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3.5  AoA  Quality,  Impact,  Speed/Duration 


CAPE  finds  the  quality  of  Army  AoAs  to  be  very  good.  While  more  concerned  about  the 
quality,  they  are  also  more  than  happy  with  the  speed  of  Anny  AoAs.  Several  interviewees 
made  the  point  that  if  the  AoA  isn’t  sufficiently  rigorous,  then  it  ends  up  being  re-done,  taking 
even  more  time. 

CAPE  highlighted  that  the  language  in  the  AoA  guidance  regarding  cost  analysis  is  boilerplate: 
“rigorous  but  not  budget  quality”.  This  may  be  an  area  for  time  saving.  There  is  a  need  to 
distinguish  between  alternatives  and  provide  information  on  cost  drivers,  but  may  not  need  all 
the  costing  work  done  up  front.  An  AoA  isn’t  about  quantity,  but  life-cycle  costs  need  an 
estimate  of  how  many  you  might  have.  Lifetime  of  missile,  for  an  example,  can  be  a  huge 
factor;  G-8  does  a  pretty  good  job  of  the  affordability  layer  on  top  of  the  AoA.  How  many  to 
buy,  the  cost  to  service  an  upgrade,  lifetime  to  service:  these  support  a  budgeting  decision.  Just  a 
rough  order  of  magnitude  is  needed  for  affordability.  Experience  has  been  that  cost  of  a  missile 
is  very  well  done  by  TRAC-White  Sands  Missile  Range  (WSMR),  but  life-cycle  costs  are  often 
done  outside  of  an  AoA.  CAPE  frequently  finds  that  life-cycle  costs  can  be  more  important  that 
the  individual  cost  of  the  missile.  The  driving  assumptions,  driven  by  the  PM  (typically  not 
analytically  derived),  will  drive  the  total  cost  of  the  program. 

CAPE's  goal  for  AoA  duration  is  9  months  or  less  from  the  first  SAG  to  Final  Brief  (not  report). 
Typically,  CAPE  oversees  one  to  two  Anny  ACAT  I  programs,  but  sometimes  three  to  four. 
When  there  is  good  communication  across  the  team,  AoAs  tend  to  be  well  defined,  with  a  clear 
set  of  alternatives  and  well-scoped  and  defined  scenarios.  When  there  is  less  robust 
communication,  the  scope  changes,  alternatives  change,  and  timelines  get  extended.  Typically, 
alternatives  will  drop  during  the  course  of  the  study.  If  the  work  is  done  right,  people  can  have 
different  opinions  about  the  trade  space. 

One  concern  highlighted  was  the  lack  of  appreciation  by  senior  leaders  of  what  it  takes  to  get 
item-level  performance  data  characteristics.  Conducting  the  research  and  development  for  new 
items  is  time  intensive.  Getting  physical  properties  from  Research  and  Development  Centers 
(RDECs)  and  the  S&T  community  for  these  things  and  putting  them  into  the  simulations  takes  a 
lot  of  time.  More  advanced  notice  is  needed  in  order  to  get  the  infonnation  required  and  the 
developmental  test  data  released. 

3.6  Study  Advisory  Group  (SAG) 


For  OSD-led  SAGs,  pre-SAG  coordination  with  OSD  CAPE  and  Office  of  the  Under  Secretary 
of  Defense  for  Acquisition,  Technology,  and  Logistics  (OUSD(AT&L))  helps  alleviate  any 
issues  or  surprises  before  the  SAG.  The  SAG  membership  will  typically  include  the  Joint  Staff, 
AT&L,  other  Services  (if  they  have  an  interest),  Army  stakeholders,  and  sometimes  combatant 
commands. 

At  the  time  of  the  interviews,  Mr.  Pete  Bechtel  chaired  the  Army-led  SAG.  However,  several 
interviewees  cited  that  50  percent  of  the  time,  Mr.  Mike  Moore,  a  GS-15,  would  actually  chair 
the  SAG.  When  the  Army  had  a  DUSA-OR,  Mr.  Walt  Hollis,  a  senior  executive,  would  chair 
the  SAGs  and  ensure  consistent  membership  and  discipline  in  the  process.  His  senior  rank, 
analytical  chops,  and  oversight  of  all  analysis  from  cradle  to  grave  of  a  program  helped  to  ensure 
consistency  in  the  process.  Several  interviewees  cited  the  need  for  the  Anny  SAG  chair  to  have 
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enough  seniority  (civilian  or  military)  to  manage  all  the  stakeholders  and  to  be  an  analyst 
themselves  so  that  they  would  understand  the  implications  of  any  scope  changes  on  the  analysis. 

3.7  Army  Requirements  Oversight  Council  /Senior  Leader  Involvement 


Prior  to  this  study,  AROC  staffing  via  paper  was  the  process  —which  was  not  optimal  for 
divining  senior  leader  concerns,  questions,  or  guidance.  More  recently,  the  “enhanced”  AROC 
has  senior  leader  involvement  in  ACAT  I/MAIS  programs.  An  AROC  may  not  (can’t)  be 
needed  for  every  program,  but  more  work  needs  to  be  done  to  find  out  what  information  both  the 
Senior  leaders  and  the  AoA  performers  need  to  make  their  decisions  or  lay  out  their  analyses. 
AoAs  need  to  have  regular  checkpoints  with  decision-makers.  The  AROC  can  provide  the 
perfonners  with  the  Service-guidance  that  they  need  to  proceed  with  the  AoA.  The  AoA  should 
be  tailored  to  meet  the  needs  of  both  DoD  and  the  Service  leaders.  Currently,  technical  analysts 
write  the  AoA  guidance.  The  AROC  should  approve  the  study  guidance  and  questions.  For  a 
program  with  OSD  oversight,  some  negotiation  may  be  required  in  order  to  ensure  that  the  AoA 
is  tractable  given  the  funding  and  timelines. 
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4  CASE  STUDIES 

4.1  Case  Study  Analysis 


Improved  Turbine  Engine  Program  (ITEP) 
(AMSAA-led,  completed  in  2014) 

Joint  Air-to-Ground  Missile  (JAGM)  AoA 
Update  (TRAC-led,  completed  in  2015) 

Long-Range  Precision  Fires  (LRPF)  AoA 
(TRADOC-led,  Completed  in  2015) 

Lower  Tier  Air  and  Missile  Defense 
Capability  (LTAMDC)  (TRAC-led, 
Completed  in  2016) 

Biometric  Enabling  Capability  (BEC  Inc  1) 
AoA  Update  (AMSAA-led,  Completed  in 
2015). 

Ground  Combat  Vehicle  (GCV)  AoA 
(TRADOC-led,  helped  inform  deferred 
investment  decision  and  S&T  work) 

Joint  Light  Tactical  Vehicle  (JLTV)  AoA 

Non  Line  of  Sight-  Launch  System  (NLOS- 
LS)  AoA 

CH-47  (Chinook)  AoA 
Comanche  AoA 

Future  Combat  Systems  (FCS)  AoA 


•  Pre-Shot  Threat  Detection  (ACAT 
1 1 1,  MCOE-led) 

•  Maneuver  Support  Vessel  (ACAT  III, 
Sustainment  COE-led) 

•  Ground  Mobility  Vehicle  (GMV)  (Study 
Plan) 

•  Armed  Aerial  Scout  (AAS) 

•  Armored  Multi-Purpose  Vehicle  (AMPV) 

•  Armed  Reconnaisance  Flelicopter  (ARH) 

•  Future  Cargo  Aircraft  (FCA) 

•  Mid-Range  Munition  (MRM) 

•  Ground  Soldier  System  (GSS) 

•  Guided  Multiple  Launch  Rocket  System 
(GMLRS) 

•  H igh  Mobility  Artillery  Rocket  System 
(HTMARS) 

AoA  Recommended  &  Provided 
AoA  Recommended 

AoA  Provided 


Figure  11.  Case  Studies. 


The  SAG  developed  a  list  of  post  -WSARA  AoAs  that  would  cover  the  gamut  based  on  ACAT 
level  and  performer  (see  Figure  1 1).  This  list  includes  some  great  success  stories  as  well  as 
AoAs  that  were  challenged  in  some  way.  The  SAG  members  provided  the  AoAs,  data  items, 
and  answers  to  a  series  of  questions  about  the  AoAs.  We  assembled  these  in  a  spreadsheet  (See 
Figure  12  for  a  G-3  Cl  Chart).  Regarding  AoA  practices,  the  Long-Range  Precision  Fires 
(LRPF)  AoA  contained  almost  every  process  element  required  in  typical  guidance  (multiple 
alternatives,  extensive  trades  analysis,  operational  effectiveness,  performance  analysis,  cost  and 
affordability  analysis,  risk  analysis,  multiple  scenarios  and  models  and  simulations  (M&S),  etc.) 
It  also  took  quite  a  bit  of  time  to  conduct  because  of  this  extensive  scope.  The  ongoing  Ground 
Mobility  Vehicle  (GMV)  AoA  is  an  example  of  an  AoA  that  was  specifically  tailored  to  address 
only  questions  of  concern  to  the  decision-maker.  Because  of  that,  the  AoA  is  greatly 
streamlined. 


Between  1  July  2013  and  8  April  2016,  TRADOC  conducted  exactly  two  ACAT  III  AoAs:  Pre- 
Shot  Threat  Detection  at  Maneuver  Center  of  Excellence  (MCoE)  (Benning),  and  Maneuver 
Support  Vessel  -  Light  at  Sustainment  CoE  (Lee).  They  began  another  one  because  regulations 
required  an  AoA  though  ARCIC  (and  the  MDA)  did  not  believe  it  needed  to  be  done  (Light 
Engineer  Utility  Trailer).  They  consequently  terminated  this  study  immediately  following  the 
study  plan  approval  briefing.  ARCIC  also  has  one  ACAT  I-like  study  that  the  MCoE  is  leading 
(Small  Arms  Ammunition  Configuration)  that  is  AoA-like  in  scope,  rigor,  and  defensibility,  but 
is  not  an  AoA. 
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In  part,  ARCIC  accomplished  this  low  AoA  rate  using  two  different,  but  complementary 
methodologies.  The  first  is  that  through  30  September  2015,  ARCIC  conducted  an  Analysis 
Assessment  of  Sufficiency  for  every  ACAT  III  CDD,  and  every  CPD,  in  order  to  determine  if 
the  existing  body  of  knowledge  is  sufficient  to  meet  the  fundamental  purpose  of  an  AoA  as 
described  above.  Of  the  ~45  Finding  Memos  issued  through  30  September  2015,  ah  but  2  found 
that  the  fundamental  purpose  of  an  AoA  had  been  met,  and  so  the  Anny  ’avoided’  conducting  an 
AoA  in  more  than  ~95%  of  cases.  [In  the  case  of  those  two,  both  requirements  had  achieved 
AROC  approval  in  2009,  and  were  components  of  an  Engineering  set  (Concrete  Saw,  and 
Asphalt  Patcher)  that  was  assessed  in  concert  with  the  MDA  that  there  were  no  infonnation 
needs  that  the  AoA  would  provide.  Consequently,  the  MDA  completed  some  market  research 
and  fielded  the  capability.] 

Beginning  1  October  2015,  ARCIC  implemented  a  new  methodology  co-developed  with  TRAC 
and  DASA-CE  to  detennine  the  appropriate  level  of  comparative  analysis  for  each  specific 
requirement.  The  information  paper  found  in  section  F-10  succinctly  communicates  the 
background,  and  shows  the  process  flow.  This  methodology  allows  the  use  almost  exclusively  of 
a  Cost  Benefit  Analysis  for  CPDs  as  the  appropriate  level  of  analysis.  The  reason  for  this  is  that 
the  analysis  required  to  support  a  CPD  must  address  quantity,  such  as  which  units  get  the 
capability,  in  which  quantities,  and  on  what  timeline.  The  Cost  Benefit  Analysis  is  ideally  suited 
for  CPD  kind  of  analyses.  With  a  CDD,  it  is  not  quite  so  simple,  as  the  supporting  analysis  must 
address  the  preferred  set  of  system  attributes  to  support  a  decision  of  whether  to  pursue 
production  and  fielding,  or  not  pursue.  Therefore,  those  cases  use  the  Analysis  of  Assessment  of 
Sufficiency  for  CDDs.  Given  their  past  -95%  rate  of  meeting  the  AoA  requirement  with  prior 
analytical  work,  ARCIC  doesn’t  anticipate  any  increase  in  ACAT  III  AoAs  in  TRADOC. 

Actually,  there  is  a  third  ACAT  III  AoA  in  TRADOC,  which  TRAC  is  perfonning;  the  Ground 
Mobility  Vehicle  discussed  above.  This  is  the  AoA  TRAC  wanted  to  use  as  a  proof  of  principle 
to  demonstrate  how  we  might  address  the  analytical  need  by  only  addressing  the  'tipping  points'. 
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4.2  Army  AoA  Case  Study  Analysis  (Recent  AoAs) 


CAPE  Oversight 

Program 

ACAT 

Date  of  MDD 

Request  Memo 

Key  Pre- 

MDD 

Activities 

Date  of 

MDD ADM 

Date  of  1st 

SAG 

Final 

Report 

Complete 

AoA 

Length 

(months) 

Final  Report 
Submitted 
to  Oversight 
Authority 

AoA 

Staffing 

(months) 

AoA 

Sufficiency 

Analytic 

Sufficiency 

Determination 

(months) 

Time  1st 

SAG  to 

Sufficiency 

(months) 

AMPV 

ID 

10/25/2011 

See  note 

3/16/2012 

4/3/2012 

8/15/2012 

4 

8/17/2012 

0 

11/21/2012 

3 

8 

LTAMDC 

ID 

5/29/2014 

2/2/2015 

1/21/2015 

12/31/2015 

11 

4/19/2016 

4 

6/30/2016 

2 

17 

LRPF 

ID 

2/28/2013 

11/6/2013 

12/4/2013 

4/30/2015 

17 

7/17/2015 

3 

9/14/2015 

2 

21 

ITEP 

ID 

4/30/2012 

11/16/2012 

12/13/2012 

6/1/2014 

18 

7/10/2014 

1 

10/1/2014 

3 

22 

JAGMIncI 

Update 

1C 

None 

11/17/2014 

12/3/2014 

6/22/2015 

7 

6/23/2015 

0 

7/20/2015 

1 

8 

IFPC  Inc  2 

1C 

11/5/2010 

8/7/2011 

11/16/2011 

4/30/2013 

17 

6/12/2013 

1 

9/13/2013 

3 

22 

GCV(MSA) 

ID 

11/3/2009 

5/11/2010 

12/15/2009 

3/24/2011 

15 

3/30/2011 

0 

7/18/2011 

4 

19 

Average 

13 

1 

3 

17 

Army  Oversight 

Program 

ACAT 

Date  of  MDD 

Request  Memo 

Key  Pre- 

MDD 

Activities 

Date  of 

MDD ADM 

Date  of  1st 

SAG 

Final 

Report 

Complete 

AoA 

Length 

(months) 

Final  Report 

Submitted 

to  Oversight 
Authority 

AoA 

Staffing 

(months) 

AoA 

Sufficiency 

Analytic 

Sufficiency 

Determination 

(months) 

Time  1st 

SAG  to 

Sufficiency 

(months) 

BORES 

II 

7/5/2012 

See  note 

9/30/2013 

12/18/2013 

2/28/2015 

14 

4/30/2015 

2 

6/1/2015 

1 

17 

H-47  Block  II 

1C 

9/29/2014 

2/20/2015 

4/6/2015 

12/1/2015 

8 

12/2/2015 

0 

1/13/2016 

1 

9 

BtC  AOA 

Update 

1A 

None 

8/6/2014 

8/8/2014 

12/1/2015 

16 

12/1/2015 

0 

12/23/2015 

1 

17 

Average 

13 

1 

1 

13 

Overall  Average 

1 

1  1 

13 

1  i  1 

2 

15 

Figure  12.  Army  AoA  Process  (Recent  AoAs). 

While  5  APPENDIX  H  has  more  detailed  information  on  the  10  most  recently  completed  Anny 
AoAs  analyzed  to  gather  insights  and  collect  statistics,  Figure  12  above  provides  some  of  the 
data  used  to  provide  statistics  on  the  AoA  duration.  While  the  chart  above  uses  the  time  from  the 
first  SAG  to  the  completion  of  the  Final  Report  as  one  measure  of  AoA  length,  it  should  be 
pointed  out  that  CAPE  considers  the  AoA  length  to  be  the  time  from  the  first  SAG  to  the  Final 
Brief. 

As  discussed  in  several  sections  of  this  report,  there  are  quite  a  few  factors  that  contribute  to  the 
length  of  a  given  AoA.  APPENDIX  H  details  some  specific  causes  found  in  our  case  studies. 

As  mentioned  in  the  introduction,  our  analysis  of  the  10  AoAs  found  the  average  study  length  to 
be  13  months  (using  the  Army’s  definition  for  duration).  The  duration  should  be  shorter  using 
CAPE’s  definition. 
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5  CHALLENGES  AND  RECOMMENDATIONS 


Capabilities  Based  Assessment 


ITEA 

Op'l  Risks 

CONOPS 

Market  Research 

Capability  Gap 
Development  & 
Prioritization 

Affordability 

DOTmLPF-P 

Assessment 

(  Pre-MDD  Activities  (Form  AoA  WG,  Review  Capability  Gaps,  Initiate 
long  lead  items  and  data/MMT  development 
(vulnerability/signature/scenario/M&S),  and  Conduct  Pre-AoA 
Measurement  Space  Workshop,  if/as  needed) 


ITEA:  Initial  Threat  Environmental  Assessment 
Mkt  Res/Aff:  Market  Research  /  Initial  Affordability  Goals 
MSW:  Measurement  Space  Workshop 
MMT:  Methods,  Models,  &  Tools 


Figure  13.  Related  AoA  Process  Activities. 


1)  Pre-ICD  AROC  Activities  and  Decisions. 

13  The  AoA  is  burdened  by  "immature"  ICDs.  The  AoA  must  then  refine  and  prioritize  capability  gaps, 
requirements,  and  concepts  of  employment.(f^  TRADOC  ensures  requirements-  and  capability-developers 
conduct  Capability  Based  Assessments  (CBA)(e.g.  CNA,  AWA,  CoL)*  to  standard.  (R2)Use  existing  processes 
to  prioritize  gaps  (e.g.  Capability  Portfolio  Review  (CPR)).  (0)  Need  to  ensure  that  TRADOC  has  the 
resources  and  capabilities  to  do  it. 

lb  Army  leadership  does  not  inform  scope  of  the  AoA.^tl)!  The  AROC  that  reviews  the  ICD  must  be  a 
decision-making  forum  that  isolates  concepts  for  refinement  and  the  scope  of  the  AoA.(JO)  The  AROCM 
must  codify  the  decisions  to  constrain  the  AoA's  scope.(0)  Cl  will  need  to  continue  to  negotiate  early  with 
CAPE  to  ensure  scope  of  AoA  is  workable. (0)  AoA  guidance  should  clearly  identify  decisions  being 
supported  and  reflect  AROC  issues. 

lc  Data,  Models,  and  Scenarios.  For  developmental  systems,  the  Army  must  develop  data  and  M&S  to 
support  analyses.  When  these  do  not  exist,  a  forecast  of  the  analysis  requirements  is  imperative. 

(^Establish  predictable  resourcing  strategy  and  AoA  forecasting  to  preclude  cold  start  of  M&S,  data  & 
scenarios  to  jump-start  AoA. 

2  Acquisition  authorities.  NDAA  2016  delegated  specific  acquisition  authorities  to  CSA;  however,  OSD 
CAPE  still  defines  AoA  guidance  and  determines  "sufficiency"  of  the  analysis.(F^  Request  NDAA  2017 
authorities  be  delegated  to  the  Army  for  ACAT  I  programs  where  the  Army  is  tne  decision  authority  :  1) 
developing  AoA  guidance  that  reflects  AROC  decision  needs  2)  determining  AoA  "sufficiency." 

3  AoA  Process  Discipline.  Staffing  timelines,  sequential  reviews,  and  mission  creep  extend  AoA  timelines. 
Continue  central  coordination  of  AoAs  that  are  planned  and  in  progress.(R2)  Codify  decisions  (AROC- 

and  SAG-)  to  minimize  mission  creep.(R|)i  Appropriate  SAG  Chair  &  Membersmp.(R3)  Conduct  parallel 
reviews  in  TRADOC,  HQDA,  and  CAPE. 


Figure  14.  Challenges  and  Recommendations. 
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The  first  three  recurring  challenges  are  under  what  we  describe  as  Pre-ICD  AROC  Activities  and 
Decisions.  This  is  the  area  believed  to  be  the  biggest  payoff  to  make  the  AoA  process  more 
efficient  and  effective. 

The  first  challenge  (labeled  la  in  Figure  13  and  Figure  14)  has  been  that  many  of  the  analytical 
products  that  should  underpin  the  ICD  and  support  the  kickoff  of  the  AoA  haven’t  been  done  or 
are  insufficient  to  inform  the  start  of  the  AoA.  The  AoA  then  has  had  to  develop  these  “in¬ 
stride”  with  the  analysis.  Organizational  and  operational  concepts  and  employment  TTP  are 
often  absent  or  immature.  The  AoA  then  concurrently  underpins  requirements  and  refines  gaps. 
Lessons  learned  show  that  weak  and/or  unsupportable  requirements  documents  and  initiation  of 
the  AoA  with  insufficient  infonnation  on  the  technology,  requirements,  and  concepts  of 
employment  to  conduct  an  evaluation  can  add  6  months  or  more  to  the  timeline.  The  first 
recommendation  is  for  TRADOC  to  ensure  that  requirements  and  capability  developers  conduct 
CBAs  to  standard.  (The  Anny  uses  CNAs,  AW  As,  and  CoLs  to  constitute  the  CBAs  with  some 
special  CBAs  done  by  the  CoEs.  A  second  recommendation  is  to  ensure  that  TRADOC  has  the 
resources  and  capabilities  needed  to  do  the  CBA,  especially  the  right  human  capital.  A  third 
recommendation  is  to  use  existing  processes  such  as  the  CPR  to  allow  Senior  Leaders  to 
emphasize  (prioritize)  gaps  to  inform  resourcing  for  CBAs.  The  acquisition  and  S&T 
communities  have  a  role  here  and  can  help.  A  prioritization  could  help  direct  their  efforts. 

A  second  perceived  challenge  (labeled  lb  in  Figure  13  and  Figure  14)  has  been  that  the  Army 
Leadership  hasn’t  had  the  opportunity  to  inform  the  scope  of  an  AoA.  While  a  paper  AROC  has 
always  been  in  use,  it  is  perhaps  not  the  best  forum  for  divining  Senior  Leader  questions  and 
scope  guidance.  Utilizing  the  revitalized  AROC  process  for  ACAT  I  programs  to  gain  Army 
guidance  early  and  nail  down  AoA  scope  in  an  AROC  Memorandum  should  greatly  infonn  the 
process.  Since  CAPE  still  maintains  oversight  of  the  AoA  guidance  and  study  plan  for  ACAT  I 
programs,  G-3  Cl  will  need  to  continue  to  negotiate  with  CAPE  early  to  ensure  that  the  AoA 
scope  remains  workable  and  guidance  reflects  AROC  issues. 

For  developmental  systems,  there  are  many  items  that  take  a  long  lead  time  to  develop  -  such  as 
vulnerability,  signatures,  threat/foreign  systems  data,  as  well  as  M&S  modifications  that  may  be 
necessary  to  model  new  behaviors  and  new  scenarios  that  need  to  be  developed  (challenge  lc). 

If  the  Anny  hasn’t  invested  in  the  previous  FY  to  get  these  things  done,  the  next  FY  will 
encounter  scheduling  delays  to  do  these.  So  a  forecast  of  the  analysis  requirements  is  imperative 
along  with  the  dedicated  funding  necessary  to  support  these  activities.  The  dedicated  funding 
now  established  to  support  TRAC/AMSAA  is  allowing  pre-MDD  activities  to  get  started. 

The  second  challenge  area  has  to  do  with  acquisition  authorities.  While  the  2016  NDAA 
delegated  some  acquisition  authorities  to  the  Service  Chiefs,  the  WSARA  still  vests  CAPE  with 
the  responsibility  for  AoAs  of  MDAPs,  regardless  of  delegation.  Without  further  authorities  for 
ACAT  I  systems,  CAPE  still  retains  authorities  to  write  guidance  and  determine  AoA 
"sufficiency”— they  still  hold  the  trump  card.  This  may  not  give  us  the  biggest  bang  for  the  buck 
in  terms  of  process  reform.  While  this  may  add  some  efficiencies  to  the  timeline,  there  are 
benefits  to  CAPE  retaining  oversight  (buy-in,  checks  and  balances,  and  better  Joint 
collaboration). 

The  final  challenge  area  has  to  do  with  discipline  in  the  AoA  process.  This  is  another  big 
potential  payoff  area.  The  first  recommendation  has  to  do  with  continuing  central  coordination 
of  AoAs  planned  and  in  progress.  The  second  is  to  codify  decisions  from  either  the  SAG  or  the 
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AROC  to  minimize  mission  creep.  Mission  Creep  —  additional  study  issues  or  desired  expansion 
of  the  analysis  representing  a  special  interest  and  not  directly  related  to  the  milestone  decision, 
can  increase  AoA  timeline  (1  to  6  months).  SAGs/AROCs  should  not  revisit  previously 
discussed  topics/decisions.  All  SAG  objectives  should  be  decision  points  and  clearly  defined 
upfront.  Having  the  appropriate  chair  and  the  appropriate  membership  has  been  a  finding  in 
many  of  the  AoA  reviews.  The  chair  must  have  sufficient  gravitas  to  maintain  control  of  the 
SAG  and  have  sufficient  analytical  skills  to  understand  the  implications  of  any  proposed  study 
questions  or  potential  expansions  on  the  analysis.  Ensuring  that  you  have  all  the  right 
stakeholders,  both  inside  and  outside  the  Army,  can  alleviate  concerns  or  delays  down  the  road. 

If  HQDA  and  TRADOC  conduct  reviews  in  parallel,  the  Anny  can  greatly  shorten  the  AoA 
staffing  timeline. 
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APPENDIX  B  REQUEST  FOR  ANALYTICAL  SUPPORT 


REQUEST  FOR  ANALYTICAL  SUPPORT 


Performing  Division:  RA 

Account  Number:  2016058 

FY:  2016 

Acronym:  AoA  PI 

Start  Date:  24-Mar- 16 

Est  Comp 1  Date:  29-Apr-16 

Title:  Analysis  of  Alternatives  (AoA)  Process  Improvement  (PI) 


Category:  Material  Systems  and  Acquisition  Resource  Analysis 

Method:  In-house 

Sponsor  (e.g.,  DCS-G3)  Name:  DCS-G8  Other 

Office  Symbol:  DAPR-ZA 

Phone: 

E-Mail:  donald.c.tison.civ@mail.mil 

POC:  Mr.  Don  Tison 

Resource  Estimates: 

a.  Estimated  Hrs:  200 

b.  Estimated  Funds: 

Models  to  be  Used: 

Product:  Briefing  and  Report 

Description/Abstract: 

This  is  a  30-day  study  to  help  the  Army  determine  how  to  shorten  the  timeline  for  accomplishing  an  Analysis  of  Alternatives  (AoA). 
AoAs  are  an  important  part  of  the  Army  acquisition  process.  However,  the  current  timelines  for  completing  an  AoA  are  a  contributing 
factor  to  the  very  long  lead  times  needed  for  Army  acquisition  programs. 


Study  Director/POC Signature:  ^  ,  ,  31  Crs.  1  Phone:  703-806-5617 

Study  Director/POC:  Ms.  Renee  G  Carlucci 

_ PART  2 _ 

Background/Statement  of  Problem: 

LTG  Murray,  Deputy  Chief  of  Staff  G-8,  requested  a  short  30-day  review  to  determine  how  to  shorten  the  AoA  timeline.  This  study 
led  by  CAA  will  include  many  of  the  Army  and  OSD  players  involved  in  the  AoA  process. 


Scope: 

The  effort  will  address:  (1)  Prioritization  of  AoAs,  (2)  Scope/Measurement  Space:  "As  Is"  process  mapping,  "To  Be”  process 
mapping.  Examination  of  historical  case  studies.  Utilization  of  TRAC's  AoA  Primer  and  previous  acquisition  reviews,  (3)  Study 
Directive/AoA  Guidance  Development,  (4)  Technical  Review  (Study  Advisory  Group  (SAG))  Process:  Requirements  and  gaps, 
Scope/Measurement  Space,  Results,  (5)  Funding,  and  (6)  Options  for  who/what  office  should  synchronize  AoAs  for  the  Army. 


Issues: 


Milestones: 


Initial  Scoping  Session,  25  March  2016. 

Weekly  AoA  PI  SAG  meetings  via  telecom  or  VTC. 

CAA  Division  Chief  Signature:  .  Octal,. 

Date  2-5  (  G 

CAA  Division  Chief  Name:  Ms.  RejXh  G  Carlucci 

Signatures 

Sponsor  Concurrence  Signature:  f  /  . 

Date  zyfid+K.  tC 

Sponsor  Name  (COL/DA  Div  Chief/GO/SES):  Mr.  Don  Tison 

Print  Dale:  25-May-16 
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APPENDIX  C  LITERATURE  RE  VIE  W/REFEREN  CES 

U.S.  Code  USC  Title  10,  Section  2366a:  Major  Defense  Acquisition  Program:  Certification 
Required  Before  Milestone  ‘A’  Approval 

Public  Law  1 1 1-23  Weapon  Systems  Acquisition  Refonn  Act  (WSARA)  (22  May  2009): 
introduced  statutory  requirements  to  conduct  AoAs  along  with  introduction  of  the  Joint  Staff  in 
reviewing  the  final  results  of  AoAs. 

National  Defense  Authorization  Act  for  Fiscal  Year  2016  (NDAA  2016):  pushed  authorities  for 
acquisition  decision  down  from  Office  of  the  Secretary  of  Defense  (OSD)  to  the  Services,  the 
NDAA  2016  also  retained  the  requirement  for  OSD  to  develop  AoA  study  guidance  and  to 
evaluate  AoAs  for  sufficiency  at  completion. 

Department  of  Defense  Directive  (DoDD)  5105.84  (dated  May  11,  2012) 

Department  of  Defense  Instruction  (DoDI)  5000.02  Subject:  Operation  of  the  Defense 
Acquisition  System  (dated  January  7,  2015) 

CJCSI  3170.011  Joint  Capabilities  Integration  and  Development  System  (23  JAN  2015) 

Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development  System  (JCIDS) 
(12  FEB  15) 

Government  Accounting  Office,  GAO-09-665,  Many  Analyses  of  Alternatives  Have  Not 
Provided  a  Robust  Assessment  of  Weapons  Systems  Options,  September  2009. 

Army  Regulation  70-1  Army  Acquisition  Policy  (Research,  Development,  and  Acquisition)  (22 
JUL  11) 

Army  Regulation  71-9  Warfighting  Capabilities  Determination  (Force  Development)  (28  DEC 
09) 

TRADOC  Regulation  71-20:  Concept  Development,  Capabilities  Determination,  and 
Capabilities  Integration  (Force  Development)  (28  JUN  13) 

Defense  Acquisition  Guidebook,  https://dag.dau.mil 

Defense  Acquisition  University  (DAU)  Analysis  of  Alternatives  (8  Dec  2015) 

“Army  Strong:  Equipped,  Trained  and  Ready”,  Final  Report  of  the  2010  Army  Acquisition 
Review  (Decker/Wagner) 

TRAC  AoA  Primer 

ARCIC  Comparative  Analysis  Assessment  Matrix  Information  Paper 
CAPE  AoA  Assessment,  August  2013 

Re-engineer  Army  Process  for  “Analysis  of  Alternatives”  Studies,  Coordination  Draft,  John 
Riente,  18  Sep  1998 

AMSAA  AoA  Lessons  Learned,  April  2016 

Analysis  of  Alternatives  (AoA)  Review,  AMSAA,  ARCIC,  TRAC,  May  2011 
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Improving  U.S.  Army  Analysis  of  Alternatives  to  Better  Address  the  Weapon  Systems 
Acquisition  Reform  Act  Of  2009,  Senior  Service  College  Fellowship,  Thomas  Stadterman,  May 
2012 

Analysis  of  Alternatives  Focused  Business  Case  Analysis  Final  Report  1.30.14,  Headquarters, 
Air  Force/Studies,  Analyses  and  Assessments  (A9),  Jan  2014 
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APPENDIX  D  KEY  DEFINITIONS/TERMS 

AIS:  Automated  Information  System 

AoA  Analysis  of  Alternatives 

AROC  Army  Requirements  Oversight  Council 

AROCM  Anny  Requirements  Oversight  Council  Memorandum 

ASARC  Army  Systems  Acquisition  Review  Council 

CAPE  Cost  Assessment  and  Program  Evaluation 

CBA  Capabilities-Based  Assessment 

CDD  Capability  Development  Document 

CJCSI  Chairman  of  the  Joint  Chiefs  of  Staff  Instruction 

CNA  Capabilities  Needs  Analysis 

CPD  Capability  Production  Document 

DAE  Defense  Acquisition  Executive 

DOTMLPF-P  Doctrine,  Organization,  Training,  Materiel,  Leadership  and  Education,  Personnel, 
Facilities,  and  Policy 

EMD  Engineering  and  Manufacturing  Development  (phase) 

ICD  Initial  Capabilities  Document 

ICDT  Integrated  Capability  Development  Team 

IPL  Integrated  Priority  List 

JROC  Joint  Requirements  Oversight  Council 

JUON  joint  urgent  operational  need 

KPP  Key  Perfonnance  Parameter 

KSA  Key  System  Attribute 

M&S  Models  and  Simulations 

MAIS  Major  Automated  Infonnation  System 

MDA  Milestone  Decision  Authority 

MDAP  Major  Defense  Acquisition  Program 

MS  Milestone 

MS  A  Milestone  A:  Approval  that  the  Materiel  Solution  Analysis  (MSA)  Phase  is  complete  and 
pennission  to  enter  the  Technology  Maturation  and  Risk  Reduction  (TMRR)  Phase. 

MS  B  Milestone  B:  Approval  that  the  TMRR  phase  is  complete  and  pennission  to  enter  the 
Engineering  and  Manufacturing  Development  (EMD)  Phase. 
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MS  C  Milestone  C:  Approval  that  the  EMD  phase  is  complete  and  permission  to  enter  the 
Production  and  Deployment  (PD)  Phase. 

MSA  Materiel  Solution  Analysis  (phase) 

NDI  Non-Developmental  Item 

TD  Technology  Demonstration 
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APPENDIX  E  HIGHLIGHTS  FROM  LITERATURE  REVIEW 
AND  ORGANIZATION  LESSONS  LEARNED 

E-l  Lessons  Learned:  TRAC,  2016 


•  Problem:  Weak  and/or  unsupportable  requirements  documents  --  resulted  in 
increased  AoA  timelines  (3  to  6  months)  to  refine  capability  gaps  and  supporting 
analyses.  Recommended  Solution:  revitalized  AROC  and  foundational 
work/requirements  development  done  to  established  standards. 

•  Problem:  Mission  Creep  -  additional  study  issues  or  desired  expansion  of  the 
analysis  representing  a  special  interest  and  not  directly  related  to  the  milestone 
decision  can  increase  AoA  timeline  (1  to  6  months).  Recommended  Solution: 
Ensure  study  is  scoped  to  address  the  specific  decision  issues  and  special  interests 
are  kept  out  of  the  study. 

•  Problem:  Staffing  Delays*  -  Redundant  staff  actions  and  staffing  in  sequence  can 
extend  AoA  timelines  (6  to  9  months).  Recommended  Solution:  New  authorities 
now  can  allow  Army  to  eliminate  some  staff  redundancy  (e.g.  when  Army  has  lead 
for  an  AoA).  Parallel  staffing  vice  sequential  can  also  reduce  the  staffing  time. 

•  Problem:  Premature  AoA  Initiation  -  initiation  of  the  AoA  with  insufficient 
information  on  the  technology,  requirements,  and  concepts  of  employment  to 
conduct  an  evaluation  could  add  6  months  or  more  to  the  timeline. 

Recommended  Solution:  Maintain  established  standards  for  conducting  Capability 
Based  Assessments  (CBAs)  and  market  research  prior  to  AoA  initiation. 


Figure  E-l.  Obstacles  to  Timely  AoAs:  TRAC,  2016. 
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E-2  Lessons  Learned:  ARL/SLAD,  2016 

•  Long  Lead  Items 

•  Vulnerability 

•  Scenario  Development 

•  Model  Development 

•  Signatures 

•  Threats/Foreign  Systems  Data  (Characteristics,  Performance,  Cost, 
Prevalence) 

•  Capability  Gap  Identification 

•  Mission  Creep-  In  part  due  to  SAG  attendance  inconsistencies  and 
revisiting  previous  SAG  decisions 

•  Data  Availability  of  Systems 

Figure  E-2.  Obstacles  to  Timely  AoAs:  ARL/SLAD,  2016. 

E-3  Lessons  Learned:  AMSAA,  Apr  2016 


•  Based  on  After-Action  Reviews  of  11  post-WSARA 
Army  AoAs 

•  Influence  study  planning  early  by  providing  inputs 
to  study  Guidance 

•  Guidance  development  should  include  study 
representatives 

•  Study  lead  determination  process  should  be 
standardized 

•  Understanding  of  Capability  Gap 
Development  to  inform  AoA  timeline 

•  Standardize  SAG  Process 

•  Reduce  study  creep  by  limiting  expanding 
guidance  at  each  SAG 

•  Alternatives  need  to  be  set  early  in  SAG 
process  (1st  or  2nd) 

•  Need  clear  community  understanding  and 
impacts  of  tasks  coming  out  of  SAGs 

•  Address  inconsistency  in  attendance  issues 
to  ensure  previous  SAG  decisions  are  not 
revisited 

•  Identify  PM  and  RDEC  Representatives  Upfront 


•  Data  collection 

•  Workshop  Participation 

•  RFIs  (when  applicable) 

Support  Pre-MDD  Analysis  of  Long  lead  Items  to 
support  emerging  AoAs 

•  Vulnerability 

•  Scenario  Development 

•  Model  Development 

•  Signatures 

•  Threats/Foreign  Systems  Data 
(Characteristics,  Performance,  Cost, 
Prevalence) 

•  Capability  Gap  Identification 

Ensure  OSD  and  Army  involvement/awareness 
outside  of  SAGs 

•  Measurement  Space  Workshop 
participation 

•  Data  Deep  Dives 

•  Capability  Gap  Reviews 


Figure  E-3.  Lessons  Learned:  AMSAA,  Apr  2016. 
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E-4  Recommendations,  Defense  Acquisition  University,  May  2012 


•  The  Army  AoA  community  and  OSD  CAPE  should  build  a  closer  working 
relationship. 

•  AoAs  should  focus  on  the  decision  choices  and  the  decision  space. 

•  AoAs  should  identify  an  achievable,  affordable,  and  operationally  relevant 
set  of  attributes. 

•  The  focus  of  foreign  systems  should  be  to  investigate  features, 
capabilities,  attributes,  feasibility,  and  cost. 

•  The  Army  should  create  a  formal  analytical  process  that  supports  MS 
decisions,  requirements  development,  materiel  acquisition,  and  portfolio 
management  throughout  the  acquisition  process. 


Figure  E-4.  Recommendations,  Defense  Acquisition  University,  May  2012. 

E-5  Lessons  Learned:  TRAC,  ARCIC,  AMS  A  A  AoA  Review,  May  2011 


•  The  resultant  exploratory  nature  of  the  tasked  AoA  is  often  at  odds  with 
its  due  date. 

•  The  lack  of  industry  input  to  the  Materiel  Solution  Analysis  (MSA)  Phase 
limits  the  relevance  of  the  knowledge  and  data  about  materiel  solutions 
(technologies)  used  in  the  MS  A  AoA  that  are  sourced  largely  from  the 
government. 

•  Full  consideration  of  NDI  or  foreign  alternative  solutions  can  be 
complicated  by  the  lack  of  authoritative  agreements  to  transfer  data,  by 
untimely  availability  and  incompleteness  of  technical  data,  and/or  by  the 
lack  of  clear  government  responsibilities  to  obtain  and  make  that  data 
available  to  the  AoA. 

•  Treating  particular  tech  demo  (TD)  candidates  offered  by  industry  as 
specified  AoA  alternatives  (an  interest  of  OSD)  may  pose  legal  issues  WRT 
source  selection.  It  certainly  complicates  timely  AoA  use  of  vetted  and 
validated  data  collected  during  the  concurrent  demo. 

•  The  numbers,  scope  of  effort  and  suspenses  of  AoAs  are  dictated  with 
little  regard  for  the  Army's  finite  capacity  to  produce  them. 


Figure  E-5.  Lessons  Learned:  TRAC,  ARCIC,  AMSAA  AoA  Review,  May  2011. 
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E-6  Lessons  Learned:  GAO  Report,  Sep  2009 


•  Reviewed  32  Programs  from  all  Services 

•  Most  of  the  programs  reviewed  either  did  not  conduct  an  AOA  or 
conducted  an  AOA  that  focused  on  a  narrow  scope  of  alternatives  and  did 
not  adequately  assess  and  compare  technical  and  other  risks  of  each 
alternative. 

•  Found  that  the  programs  that  considered  a  broad  range  of  alternatives 
tended  to  have  better  cost  and  schedule  outcomes  than  the  programs 
that  looked  at  a  narrow  scope  of  alternatives. 

•  Factors  limiting  the  effectiveness  of  AOAs  and  their  ability  to  identify  the 
most  promising  option  and  contribute  to  a  sound  business  case  for 
starting  a  weapon  system  program:  (1)  service  sponsors  lock  into  a 
solution  early  on  when  a  capability  need  is  first  validated  through  DOD's 
requirements  process  and  before  an  AOA  is  conducted;  (2)  AOAs  are 
conducted  under  compressed  time  frames  in  order  to  meet  a  planned 
milestone  review  or  fielding  date  and  their  results  come  too  late  to  inform 
key  trade  off  decisions;  and  (3)  DOD  does  not  always  provide  guidance  for 
conducting  individual  AOAs. 


Figure  E-6.  Lessons  Learned:  GAO  Report,  Sept  2009. 
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E-7  Decker-Wagner  (Army  Acquisition  Review)  Findings,  Jan  2011 


•  The  mean  time  to  approve  an  Acquisition  Category  (ACAT)  I  system  requirement  is  15  months  with  an 
ACAT  II  taking  22  months  and  an  ACAT  III  taking  18. 

•  Common  causes  of  major  program  cancellation  include: 

•  Overly  optimistic  forecast  of  funding  available  for  Army  modernization. 

•  Weak  baseline,  modeling,  trade  studies  or  analysis  of  alternatives. 

•  Unconstrained  weapon  system  requirements. 

•  Underestimation  of  risk,  particularly  technology  readiness  levels. 

•  Failure  to  eliminate  technological  risk  prior  to  Milestone  B  (MS  B)  approval. 

•  Program  skipped  or  under-resourced  pre-MS  B  prototyping. 

•  Too  many  programs  started  only  to  prove  unaffordable  in  the  budget  and  Future  Years  Defense 
Program  (FYDP). 

•  Affordability  reprioritization. 

•  Schedule  slip. 

•  Requirements  and  technology  creep. 

•  Cost  overruns. 

•  Program  restructured,  quantities  cut,  unit  costs  skyrocketed  and  program  support  lost. 

•  Solutions: 

•  Make  Requirements  Process  Collaborative  and  Timely  -  Army  Regulation  (AR)  71-9  provides  for 
collaborative  requirements  development  with  an  Integrated  Capabilities  Development  Team 
(ICDT).  Unfortunately,  TRADOC  has  no  authority  to  require  participation,  but  can  only  “invite" 
those  who  choose  not  to  participate  and  will  later  critique  the  requirement. 

Figure  E-7.  Decker-Wagner  (Army  Acquisition  Review)  Findings,  Jan  2011(1  of  2). 

A  TRADOC-led  Integrated  Capabilities  Development  Team  (ICDT)  with  personnel  from  the 
Anny  Staff  (ARSTAFF)  and  Secretariat,  Anny  Materiel  Command  (AMC),  Army  Test  and 
Evaluation  Command  (ATEC)  and  other  Army  Commands  should  collaboratively  develop 
requirements  documents  for  AROC  approval  of  most  programs:  -  Amend  Army  Regulation 
(AR)  71-9  to  give  the  TRADOC  commanding  general  (CG)  the  authority  to  task  non- TRADOC 
organizations  for  ICDT  participation.  ICDT  representatives  must  have  the  authority  to  speak  for 
and  commit  their  organizations. 

TRADOC  and  Army  requirements  approval,  MDD,  MS  A  and  MS  B  decisions  must  be 
synchronized  to  align  with  the  Army  POM  and  budget  development  schedules. 

The  study  authors  found  the  Army’s  documented  reasons  for  cancellation  to  be  too  general  and 
in  conflict  with  the  facts  as  they  knew  them  based  on  personal  experience  with  many  of  the  22 
programs  and  discussions  with  others  in  the  Anny  who  had  worked  on  the  programs.  There  are 
typically  multiple  causes  for  each  program  cancellation,  and  for  each  program  conflicting 
explanations.  Although  there  are  many  different  causes  that  contribute  to  the  cancellation  of  a 
program,  the  cancelled  programs  often  shared  several  of  the  same  problems.  A  few  were 
cancelled  because  the  threat  disappeared.  Yet  the  reason  most  often  cited  for  program 
cancellation  was  tersely  described  as  a  ‘change  in  priorities’  or  ‘affordability’. 
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•  Conduct  competitive  prototyping  at  the  component,  subsystem  and  even  system  level  prior  to  EMD 
to  reduce  technical,  schedule,  cost  and  performance  risk. 

•  Increase  the  number  of  qualified  systems  engineering,  cost  estimating,  quality  assurance  and  ORSA 
(military  FA  49  and  civilian  1515)  personnel  in  Army  acquisition. 

•  Leverage  FFRDCs  and  UARCs  to  make  up  for  the  shortfalls  in  the  Army's  systems  engineering,  quality 
assurance  and  analytic  capabilities  until  the  bench  is  replenished. 

•  Increase  both  AMSAA  and  TRAC  funding  by  $10M  per  year  to  conduct  AoAs. 

•  Continue  to  resource  the  DA  program  for  data  collection  and  development  of  scenarios,  models  and 
simulations  to  support  requirements  development. 

•  Develop  needed  analytic  portfolio  management  tools  for  the  G-8  and  CPRs. 

•  In  the  past,  the  analytic  support  and  backup  for  the  VCSA's  and  other  Army  Staff  (ARSTAFF)  reviews 
and  decision  bodies  came  from  the  Deputy  Under  Secretary  of  the  Army  (Operations  Research) 
(DUSA(OR)),  but  this  position  was  eliminated  in  2006.  The  DUSA(OR)  was  the  spokesman  for  Army  studies, 
analyses  and  testing  within  the  Army  and  with  DoD. 

•  With  the  elimination  of  this  position,  the  Army  lost  its  single  independent  voice  across  the  Army,  in 
the  JCS,  at  OSD  and  on  the  Flill  for  ORSA,  requirements,  prototyping,  experimentation,  and  testing. 

•  The  root  causes  for  troubled  and  terminated  programs  usually  stem  from  the  developmental  planning 
period  from  before  MDD  to  Milestones  A&B. 

•  In  addition  to  a  timely  and  productive  requirements  development  capability,  Army  acquisition  must 
have  its  own  organic  capability  to  lead  pre-Milestone  A  advanced  systems  concept  formulation,  explore 
promising  advanced  technology  concepts,  and  conceive  and  advocate  advanced  programs  before  there  is 
a  Program  Manager  assigned; 


Figure  E-8.  Decker-Wagner  (Army  Acquisition  Review)  Findings,  Jan  2011(2  of  2), 

Numerous  acquisition  studies  and  DoD  directives  have  recommended  competitive  prototyping  at 
the  component,  subsystem,  and  even  system  level  prior  to  EMD  to  reduce  technical,  schedule, 
cost,  and  performance  risk.  Pre-EMD  subsystem  and  system  prototyping  were  a  major  benefit  in 
many  of  the  successful  programs  studied.  Unfortunately,  acquisition  strategies  too  often  omit 
this  in  order  to  shorten  the  schedule  and  lower  development  cost,  only  to  result  in  more 
development  time  and  cost  due  to  technical  problems  during  EMD  that  could  have  been 
prevented  with  competitive  prototyping.  Similarly,  during  development  many  programs  do  not 
invest  sufficiently  to  reduce  eventual  life  cycle  costs. 
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E-8  The  Joint  Capabilities  Integration  and  Development  System  (JCIDS) 
Review  -  Goldwater  Nichols  Analysis  Working  Group 


Requirements  Reform  Recommendations 

^ 

□  Recommendation:  Establish  core  elements  required  for  developing  Capability-Based  Assessments 
(CAPE-led  Analysis  Working  Group) 

■  Bolster  up-front  analytical  rigor  to  derisions  imparting  materiel  solutions 

•  Core  elements:  study  advisory  group  with  external  stakeholders  charged  with  ensuring  analytic 
sufficiency  by  issuing  study  guidance,  vetting  threat  projections,  approving  study  plans,  and 
identifying  affordability  issues,  as  appropriate 

□  Recommendation:  “Develop  Middle  Tier”  Requirements  Lane  (JS-led  Working  Group) 

•  Support  CCMD  needs  which  don’t  meet  urgency  criteria  or  fielding  timelines  of  JUON/JEON  to 
“middle  tier” 

•  Develop  Rapid  Requirements  Document:  hybrid  between  JI  O.N  and  ICD  (content  and  staffing); 
eliminate  JEON;  enables  rapid  prototyping/rapid  fielding  to  meet  “emergent  military  needs” 

•  Develop  Agile-Capability  Development  Document:  leverage  the  Information  Systems-Capability 
Development  Document  model  to  streamline  requirements  process,  delegate  requirements 
approval  authority 

□  Recommendation:  Analytical  Expertise  of  the  Requirements  Community  (JS/CAPE  Working  Groups) 

•  Expanded  human  capital  is  needed  to  improve  analytical  expertise  for  the  identification, 
assessment,  aud  approval  of  requirements 

•  National  Defense  Authorization  Acts  2015, 2016  and  engagements  with  Congress  reveal  intent  to 
expand  the  analytical  rigor  to  requirements,  acquisition  and  resourcing  determinations  with 
specific  emphasis  in  operations  research  and  systems  engineering 


Figure  E-9.  Goldwater  Nichols  Requirements  Reform,  2  Mar  16. 

The  JCIDS  Review  Goldwater  Nichols  Analysis  Working  Group  was  comprised  of  members 
from  the  Joint  Staff,  the  Services,  CAPE,  AT&L,  and  Director,  Operational  Test  and  Evaluation 
(DOT&E).  The  groups’  goal  was  to  set  a  strong  foundation  to  enable  effective  acquisition 
program  development  and  progress  toward  successful  production  and  fielding.  Their  focus  was 
on  optimizing  analytic  rigor  in  the  CBA  and  other  studies  used  to  identify  capability 
requirements  and  gaps.  Their  approach  was  to  establish  a  CBA  process  baseline,  conduct  a  case 
study  review  of  recent  CBAs  (CAPE  reviewed  nine  CBAs),  and  to  determine  lessons  learned, 
good  practices,  and  pitfalls  to  avoid.  The  figure  above  highlights  the  key  recommendations  from 
the  effort.  However,  many  of  their  observations  echo  findings  from  our  AoA  review  (e.g.,  some 
CBAs  were  insufficient  for  informing  ICDs  and  supporting  AoAs).  They  recommend  the  use  of 
external  stakeholders  to  help  improve  CBAs,  providing  better  ICDs  and  AoAs  and  to  help  reduce 
CBA/AoA  cycle  times  and  the  use  of  SAGs  as  are  used  in  AoAs.  They  caution  against 
specifying  performance  attributes  too  narrowly  or  without  analytical  justification,  which  can 
limit  trade  space. 
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APPENDIX  F  PROCESS  MAPS 

F-l  Integrated  Army  Decision  Support  System 
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F-2  Deliberate  Requirements  Process 


Figure  F-2.  Deliberate  Requirements  Process. 
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F-3  Process  as  Designed — AC  AT  I  Programs 
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Figure  F-3.  Process  as  Designed 
ACAT  I  Programs. 
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Figure  F-4.  Process  as  Designed-  Additional  Decision  Points 
ACAT  I  programs. 
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F-4  Capabilities  Needs  Analysis  (CNA)  Process 
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s> 


□  Strategic  Planning  Guidance 

□  AWFC  Running  Estimates 

□  Six  Scenarios 

□  Army  Concept  Framework  (RCs) 

□  Operational  &  Organizational 
Concepts 
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Figure  F-5.  Capabilities  Needs  Analysis  Process. 

The  key  inputs  to  the  CNA  process  include:  scenarios,  concepts  (specifically  the  required 
capabilities),  operational  and  organizational  concepts  as  they’re  developed. 

The  CNA  objective  was  established  to  meet  the  ARCIC  Director’s  intent.  It  was  developed 
considering  the  context  of  the  Army  Operating  Concept,  Army  Warfighting  Functional  Concepts 
on  how  to  fight  in  the  future,  and  five  Support  to  Strategic  Analysis  compliant  scenarios  to 
enable  prioritization  of  Resourcing  and  Developments  through  a  capability-based  analysis  across 
Warfighting  Functions  as  well  as  Formations.  This  objective  was  decomposed  into  3  issues  and 
1 1  essential  elements  of  analysis  to  help  understand  the  extent  of  the  problem  set  needed  to 
answer  the  CNA  objective.  In  essence,  the  CNA  must  determine  preferences  in  three  areas. 

First,  determine  the  ordered  preference  of  which  of  the  current  programs  to  sustain  in  the  force 
or  programmed  budget  to  ensure  the  Army  meets  its  joint  warfighting  requirements  and  which 
can  be  traded-off  with  minimal  risk  to  enable  procuring  a  new  capability  that  mitigates  or  solves 
a  high  risk  gap.  Second,  determine  which  of  the  tasks  representing  the  Army’s  Warfighting 
Requirements  cannot  be  achieved,  describe  the  capability  gap  that  results  and  assess  the 
operational  risk  of  that  gap  in  the  scenario  set.  This  provides  an  ordered  preference  for  which 
gaps  to  address  soonest.  Third,  determine  an  ordered  preference  for  the  solutions  that  solve  or 
mitigate  the  gap  to  enable  future  investment  focus. 
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So  then  the  next  step  was  to  detennine  how  to  represent  all  of  these  competing  objectives  in  a 
cohesive  analysis  that  would  account  for  all  stakeholder  and  Force  Modernization  Proponents 
equities. 

F-5  High  Level  Process  Map  (ICD  to  MDD) 


What  are  the  analytical  activities  supporting  requirements  development? 

Source:  “Army  Strong:  Equipped,  Trained  and  Ready”,  Final  Report  of  the  2010  Army  Acquisition  Review  (Decker/Wagner) 


Figure  F-6.  High  Level  Process  Map  (ICD  to  MDD). 
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F-6  High  Level  Process  Map  (MDD  to  MS  A)) 
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Figure  F-7.  High  Level  Process  Map  (MDD  to  MSA). 
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F-7  High  Level  Process  Map  (MS  A  to  MS  B) 
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Source:  “Army  Strong:  Equipped,  Trained  and  Ready”,  Final  Report  of  the  2010  Army  Acquisition  Review  (Decker/Wagner) 


Figure  F-8.  High  Level  Process  Map  (MS  A  to  MS  B). 
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F-8  AoA  Related  Process  Activities 
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Figure  F-9.  AoA  Related  Process  Activities. 
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F-9  AC  AT  I  AoA  Process  Steps 


INDEX  Input 

Activity 

Lead  Agency 

Output 

Comments 

Predecessor 

Duration 

0  CBA/CNA  — >ICD 

AoA  work  planning  Forecast 

DAMO-CI 

Develop  proposed  work  plan  for  AoAs  coordinated 
with  Study  Agencies 

Allows  Study  Agencies  to  do  initial 
work  planning  forforecast  AoAs. 

Approved  CBA 

Quarterly 

1  CBA/CNA  — >ICD 

AROC/JROC 

TRADOC  CoE/DAMO-CI 

ICD  Approval 

Provides  the  foundation  for  all  future 
analysis/assessments 

Approved  CBA 

2  AROC/JROC  Decision 

MDD  Request 

DAMO-CI/ASA(ALT) 

MDD  Scheduled 

0, 1 

1-3  mo 

3  MDD  Request 

AoA  Guidance 

Developed/Finalized  (signed) 

DCAPE/DAMO-CI 

Guidance  Directive/Document 

Approved  at  the  MDD,  sets  initial  scope 
of  the  study 

2 

1-3  mo 

4  MDD  Request 

AoA  Study  Plan  Development 

TRAC/  A  MS  AA/ AoA  Lead 
Agency 

Executive  Level  Study  Plan  (i.e.,  10  pager  required 
in  guidance) 

Done  in  conjunction  and  coordination 
with  AoA  guidance  development; 
executive  level  plan 

0,2 

1-3  mo 

5  MDD  Request 

Affordability  Cost  Goals 
Developed 

G-8 

Cost  Goals  input  to  Study 

Provides  APUC,  lifecycle  training  and 
sustainment  goals 

2 

1-3  mo 

6  AoA  Guidance 

MDD  Decision 

DA  E/A  A  E 

Authority  to  enter  MSA  Phase  and  conduct  AoA 

ASARC  Memo  or  ADM 

3,4,5 

lmo 

7 

Capability  Gap  Refinement 

TRAC/Proponent 

CoE/TCM 

Understanding  and  refinement  of  the  capability 
gaps  to  enable  analysis  of  means  to  mitigate  the 

gaps 

ICD,  This  is  done  routinely  as  a  first  step 
in  AoAs,  but  should  be  done  in  the  CBA 

process 

3 

l-6mo 

8 

Measurement  Space  Drill 

AoA  Lead  Agency/  TRAC 

Methods,  Models,  Tools,  and  Scenarios  identified 
to  conduct  the  study,  full  study  team  identified 
(roles/responsibilities),  tasks  issued,  schedule 
developed,  study  scoped  to  tipping  point  issues 

Provides  Key  Input  for  Detailed  study 
plan  and  execution  plan.  Foundation 
for  AoA  data/alternative  RFI 

3, 4,5, 7 

1-3  wks 

9 

Study  Plan  Approval  SAG  IPR 1 

DCAPE/DAMO-CI 

Approved  Plan 

Approve  Study  Plan  and  Alternative 
Screening  Criteria 

8 

1-3  wks 

10 

Identify  Data  Needs/Collect 
Screening  Data 

AoA  Lead  Agency/PM 

Data  Collection  to  support  Alternative  Screening 
and  AoA  Assessment 

If  required 

9 

1-3  mo 

11 

Technology  market  research  / 
AoA  RFI 

PEO/PMs 

Range  of  candidates  and  data 
(characteristic/cost/performance  etc.)  for  use  in 
the  AoA 

Some  candidates  may  be  provided  in 
the  guidance  document,  but  most  often 
the  Study  team  is  tasked  with 
developing  potential  candidates  to  be 
evaluated. 

4,10 

1-3  mo 

12 

Screening  Analysis 

TRAC/AMSAA 

Screens  AoA  alternatives  to  feasible  set  to 

consider  in  the  AoA 

Screening  criteria  is  approved  at  the 
study  plan  SAG 

9,11 

1-3  mo 

13 

SAG  IPR  2 

DCAPE/DAMO-CI 

Approval  of  Alternatives,  provide  guidance  to 
Study  Team 

Candidates  that  fail  screening  may  be 
retained  in  the  analysis  as  special 
interest  items  from  the  SAG  members 

12 

1-3  wks 

14 

Performance  Data  Development 

AMSAA 

Develops  technical  data  to  represent  the  AoA 
alternatives 

The  earlier  developed  the  better. 

8,13 

3-9  mo 

15 

Method/Model/Tool  (MMT) 
Development 

TRAC/AMSAA 

Adjusts  existing  MMT  for  AoA  use,  create  new 
tools  for  the  AoA,  refine  methods  for  AoA 

Should  contribute  to  an  AoA  RFI 

0,9 

1-6  mo 

16 

Scenario  Development 

TRAC 

Adjusts  existing  scenarios  for  the  AoA,  develops 
new  vignettes  and  scenarios  as  needed 

0,9 

1-6  mo 

Figure  F-10.  ACAT  I  AoA  Process  Activities  (1  of  3). 
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INDEX 

Input 

Activity 

Lead  Agency 

Output 

Comments 

Predecessor 

Duration 

17 

Cost  Data  Development 

DASA-CE/TRAC 

Define  cost  analysis  rules  and  data  development 
rules  for  consistency  across  AoA  Alternatives 

Should  contribute  to  an  AoA  RFI 

0,9 

3-6  mo 

18 

Operational  Data  Development 

TRAC/Proponent 

CoE/TCM 

Refine  existing  operational  concepts/TTP  or 
develop  representation  of  future  operational 
concepts/TTP  in  MMT 

0,9 

1-3  mo 

19 

Performance  Analysis 

AMSAA 

Item  level  analysis 

14,15 

3-9  mo 

20 

Operational  Benefit  Analysis 

TRAC 

Operational  and  Tactical  benefits  of  Alternatives 
with  associated  risks 

14,15 

3-18  mo 

21 

Industrial  Base  Assessment 

AMRDEC 

Assessment  of  existing  technology  and  the 
industrial  base's  ability  to  support  each  alternative 

This  is  an  optional  study  area  — 
contingent  on  approved  study  guidance 

9,13 

1-6  mo 

22 

SAG  1 PR  3 

DCAPE/DAMO-CI 

Review  Results  and  Provide  guidance  to  Study 

Team 

Should  be  event  driven.  Could  have 
more  than  performance  and  Ops 
analysis  complete.  Should  include  a 
TRADES  plan  or  another  IPR  SAG  to 
approve  trades.  New  work  could  be 
directed  by  the  SAG 

19,20 

1-3  wks 

23 

Life  Cycle  Cost  Analysis 

DASA-CE/TRAC 

Fully  Burdened  cost  of  Ownership  of  the  AoA 
Alternatives 

13,17 

6-12  mo 

24 

Schedule  Risk  Assessment 

AMSAA 

Evaluation  of  the  proposed  Acquisition  schedules 
and  associated  risk,  Evaluation  of  technology 
maturity  and  associated  risks  of  the  alternatives 

Schedule  Risk  analysis  and  Technical 

Risk  analysis  are  now  combined  under 
Schedule  Risk  analysis. 

9,13 

3-6  mo 

25 

Performance  Risk  Assessment 

AMSAA 

Risk  of  each  alternative  achieving  proposed  level 
of  performance 

19,20 

3-6  mo 

26 

Trades  Analysis 

TRAC/A  MS  A  A 

Modified  Alternatives  to  mitigate  the 
cost/schedule/performance/operational  risks 

19,20,23,24,25 

1-3  mo 

27 

Fully  Burdened  Cost  of  Energy 
Analysis 

DASA-CE/TRAC/A  MSAA 

Energy  consumption  and  Energy  costs  for 
Alternatives 

14,18,20,26 

1-3  mo 

28 

Sustainment/Reliability  Analysis 

TRAC/A  MS  A  A 

Sustainment,  Reliability,  maintenance  manpower 
comparisons  of  Alternatives 

9,14,15 

1-6  mo 

29 

Affordability  Analysis 

G-8 

Comparison  of  Alternatives  Portfolio  Affordability 

23,26 

1-3  mo 

30 

SAG  IPR4 

DCAPE/DAMO-CI 

Final  Results  SAG 

Determine  that  all  study  issues  have 
been  addressed  or  issue  new  guidance 
to  study  team  for  additional  analysis 

23,24,25,26,27, 

28,29 

1-3  wks 

31 

Analysis  Integration 

AoA  Lead  Agency/Study 
Team 

Integrating  all  analysis  into  an  integrated  package 
to  discuss  cost/effectiveness/risks/  and  trades 

30 

1-3  mo 

Figure  F-ll.  ACAT  I  AoA  Process  Activities  (2  of  3). 


INDEX 

Input 

Activity 

Lead  Agency 

Output  Comments 

Predecessor 

Durati 

32 

Develop  Report 

AoA  Lead  Agency 

Formal  Documentation  of  all  works  into  an 

Executive  Summary,  Report,  and  supporting 
appendices/annexes 

30 

1-3  mo 

33 

Deliver  Report  to  TRADOCHQ 
for  Staffing 

When  TRAC  is  Lead 

Agency 

Review  for  Leadership  approval  to  provide  to 

HQDA 

32 

1-3  wk 

34 

Deliver  Report  To  HQDA  for 

Staffi  ng 

AoA  Lead  Agency 

Review  for  Leadership  approval  to  provide  to 

DCAPE  or  conduct  sufficiency  review  if  at  HQDA 
level 

33 

1-3  mo 

35 

Deliver  Report  to  DCAPE  for 

Staffi  ng 

DAMO-CI 

Review  for  sufficiency  and  advise  to  MDA  If  required 

34 

1-3  mo 

36 

Sufficiency  Review 

OSD  CAPE  /  HQDA 

Formal  memo  documenting  sufficiency  (30  days 
prior  to  milestone  decision) 

34  or  35 

1-6  mo 

37 

AROC/JROC  review 

DAMO-CI/G-8 

Informational  review  wrt  how  did  the  analysis 
support  requirements  changes/adjustments 

36 

1  wk 

38 

ASARC/DAB 

DAE/AAE 

Decision  to  continue  to  nest  Milestone  phase  of 
the  Acquisition  process 

37 

1-3  wk 

Figure  F-12.  ACAT  I  AoA  Process  Activities  (3  of  3). 
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F-10  AC  AT  III  Analysis  Process 


Comparative  Analysis  Assessment  Process 


S> 


Source:  “ARCIC  Comparative  Analysis  Assessment  Process  Information  Paper,  17  Feb  16 


Figure  F-13.  ACAT  III  Analysis  Process. 

ARCIC  Studies  and  Analyses  Division  (S&AD)  is  designated  as  the  ARCIC  Director’s  executive 
agent  for  proponent  Cost  Benefit  Analysis  review  and  quality  assurance. 

TRADOC  will  meet  the  spirit  of  regulatory  requirements  regarding  analysis  by  determining  and 
directing  the  appropriate  level  of  alternative  comparison,  or  comparative  analysis,  for  every 
requirement  and  potential  program  for  which  TRADOC  is  delegated  analysis  oversight 
responsibilities. 


TRADOC  will  interpret  the  regulatory  analytic  requirement  as,  “Every  acquisition  program 
requires  an  appropriate  level  of  comparative  analysis,”  for  ACAT  III  designated  programs.  This 
may  be  an  AoA,  Business  Case  Analysis,  Cost  Benefit  Analysis,  or  some  other  appropriate 
defensible  analysis. 


ARCIC  S&AD,  in  concert  with  TRAC  and  DASA-CE,  developed  the  ARCIC  Comparative 
Analysis  Assessment  Process  (see  Figure  above)  designed  to  determine  and  /  or  direct  the 
appropriate  level  of  analysis:  to  foster  early  standards  engagement;  to  ensure  the  analysis  is 
focused  appropriately  to  address  necessary  requirements  development;  and  to  meet  MDA- 
defined  data  needs  at  program  acquisition  decision  points.  This  process  is  designed  to  reduce  the 
volume  of  re-work  required  to  deliver  defensible  analysis. 


TRADOC  organizations  will  engage  S&AD  as  early  as  practical  in  their  requirement 
development  process  to  support  an  early  determination  of  the  required  type  of  analysis  to  support 
their  topic  specific  requirement.  S&AD  will  also  assist  proponents  with  developing  defensible 
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analysis  that  supports  Capability  Developer  requirement  refinement,  and  to  inform  the  MDA  at 
the  MDD,  and  later  Milestone  Decisions. 


INFORMATION  PAPER 


ATFC-ZCA  16  FEB  16 

SUBJECT:  ARCIC  Comparative  Analysis  Assessment  Matrix 

1.  Purpose.  Communicate  ARCIC’s  Comparative  Analysis  Assessment  Matrix  application. 

2.  References: 

a.  USC  Title  10,  Section  2366a:  Major  Defense  Acquisition  Program:  Certification 
Required  Before  Milestone  ‘A’  Approval 

b.  DoDI  5000.02  Subject:  Operation  of  the  Defense  Acquisition  System  (7  JAN  15) 

c.  Manual  for  the  Operation  of  the  Joint  Capabilities  Integration  and  Development 
System  (JCIDS)  (12  FEB  15) 

d.  Army  Regulation  70-1  Army  Acquisition  Policy  (Research,  Development,  and 
Acquisition)  (22JUL11) 

e.  Army  Regulation  71-9  Warfighting  Capabilities  Determination  (Force  Development) 
(28  DEC  09) 

f  TRADOC  Regulation  71-20:  Concept  Development,  Capabilities  Determination,  and 
Capabilities  Integration  (Force  Development)  (28  JUN  13) 

3.  Facts. 

a.  An  Analysis  of  Alternatives  (AoA)  is  a  statutory  requirement  for  ACAT I  programs  and 
is  a  regulatory  requirement  for  ACAT  II  and  III  programs  that  mirrors  statutes. 

b.  The  AoA  fundamentally  has  two  specific  purposes:  to  provide  combat  developers 
(CAPDEVs)  with  defensible  analytic  evidence  identifying  the  preferred  set  of  system 
attributes  that  mitigate  identified  capability  gaps,  while  still  meeting  DA  derived 
affordability  goals;  and,  to  inform  the  acquisition  Milestone  Decision  Authority  (MDA). 

c.  TRADOC  and  ASA(ALT)  coordinated  legal  reviews  confirmed  that  there  is  no 
language  in  any  U  S.  Army  or  DoD  regulation  that  addresses  an  AoA  waiver  option, 
or  waiver  authority,  for  ACAT  II  or  III  potential  programs. 

d.  On  30  DEC  09,  the  VCSA  and  USA,  directed  that  any  unfunded  requirement  and  new 
or  expanded  program  proposals  be  accompanied  by  a  thorough  Cost-Benefit 
Analysis  (C-BA).  A  27  DEC  10  SECDEF  directive  further  stipulated  that  any  new 
proposal  or  initiative  . . .  come  with  a  cost  estimate. 

3.  Background 

a.  TR  71-20  (28  JUN  13)  "Force  Development  -  Concept  Development,  Capabilities 
Determination,  and  Capabilities  Integration"  Section  2-10.  Dir,  ARCIC,  para  b. 
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SUBJECT:  ARCIC  Comparative  Analysis  Assessment  Process 

‘Approve  cost-benefit  analyses  (C-BA)  conducted  by  force  modernization 
proponents. " 

b.  ARCIC  Studies  and  Analyses  Division  is  designated  as  the  ARCIC  Director's 
executive  agent  for  proponent  C-BA  review  and  quality  assurance. 

c.  TRADOC  will  meet  the  spirit  of  regulatory  requirements  regardng  analysis  by 
determining  and  directing  the  appropriate  level  of  alternative  comparison,  or 
comparative  analysis,  for  every  requirement  and  potential  program  for  which 
TRADOC  is  delegated  analysis  oversight  responsibilities. 

d.  TRADOC  will  interpret  the  regulatory  analytic  requirement  as,  “Every  acquisition 
program  requires  an  appropriate  level  of  comparative  analysis,’  for  ACAT  III 
designated  programs.  This  may  be  an  Analysis  of  Alternatives,  Business  Case 
Analysis,  Cost  Benefit  Analysis,  or  some  other  appropriate  defensible  analysis. 

e.  S&AD,  in  concert  with  TRAC  and  DASA-CE,  developed  the  ARCIC  Comparative 
Analysis  Assessment  Process  (see  Figure  below)  designed  to  determine  and  /  or 
direct  the  appropriate  level  of  analysis:  to  foster  early  standards  engagement;  to 
ensure  the  analysis  is  focused  appropriately  to  address  necessary  requirements 
development;  and  to  meet  MDA-defined  data  needs  at  program  acquisition  decision 
points.  This  process  is  designed  to  reduce  the  volume  of  re-worfc  required  to  deliver 
defensible  analysis. 

4.  Way  Ahead:  TRADOC  organizations  will  engage  S&AD  as  early  as  practical  in  their 
requirement  development  process  to  support  an  early  determination  of  the  required  type  of 
analysis  to  support  their  topic  specific  requirement.  S&AD  will  also  assist  proponents  with 
developing  defensible  analysis  that  supports  CAPDEV  requirement  refinement,  and  to 
inform  the  Milestone  Decision  Authority  (MDA)  at  the  Materiel  Development  Decision 
(MDD),  and  later  Milestone  Decisions. 

PREPARED  BY:  COL  Thomas  Dillingham  /  S&AD  /  501-5312 
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F-ll  AC  AT  III  AoA  Process  Activities 
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Notes: 

1 .  Conduct  comparative  analysis  assessment  (AA)  to  determine  analytic  sufficiency  (whether  previous  analytic  evidence  sufficient  to  meet  intent  of  AoA).  If  sufficient,  will  publish 
AA  Findings  Memo. 

2.  Simultaneous  Affordability  Cost  Goal  Development  Request  to  DA  G8. 

3.  Capability  Gap  Refinement  (as  needed);  Literature  Review;  T echnology/Market  Research;  Screening  Analysis;  Life-Cycle  Cost  Analysis;  Performance  Analysis,  Operational 
Effectiveness  Analysis;  Risk  (technical/schedule)  Analysis;  Cost/Performance/Schedule  T rades  Analysis;  RAM  Analysis;  Affordability  Analysis;  Integration 

4.  S&AD  Review  and  staffing  with  stakeholders  for  comment;  comment  adjudication;  preparation  for  release  through  Dir,  ARCIC  (cc  CG  TRADOC)  to  DAMO-CI. 


Figure  F-14.  ACAT  III  AoA  Process  Activities. 
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APPENDIX  G  SAG  EXECUTIVE  SUMMARIES 


•  Funding:  an  assumption  will  be  that  the  Army  will  require  $10M  in  funding  for  AoAs  in 
perpetuity.  (Note:  TRAC  points  out  that  it  can  be  more  like  $20M,  but  they  have 
accomplished  using  mission  funding.)  Delays  in  distributing  AoA  funding  has  not  only  led  to 
delays  in  accomplishing  AoAs,  but  has  resulted  in  under-execution  and  marks  taken.  The 
solution  will  be  to  de-link  AoA  funding  from  AoA  prioritization  and  to  provide  dedicated 
funding  to  TRAC  and  AMSAA.  Funding  can  be  shifted  between  performers  as  needed.  Need 
to  work  with  FM&C  to  work  the  rapid  funding  for  FY16. 

•  LTG  Murray's  original  question  was  how  to  shorten  the  AoA  timelime;  all  of  the  participants 
agreed  that  the  problem  statement  needs  to  be  revised.  The  study  should  encompass  ACAT  I 
as  well  as  ACAT  II  &  III  AoAs,  acknowledging  the  differences  between  the  statutory 
requirements  for  ACAT  I  and  the  regulatory  requirements  for  ACAT  II  &  III.  The  Army's  needs 
are  different  from  CAPE's  for  ACAT  I  programs. 

•  The  study  must  look  at  improving  the  SAG  process  with  the  first  SAG  used  to  narrow  down 
the  requirements.  Need  up-front  mission  analysis  and  up-front  Senior  Leader  guidance  from 
both  Army  and  OSD  leadership  -  before  proceeding  on  the  AoA.  Number  of  alternatives  and 
additional  tasks  added  throughout  AoA  process  drive  the  scope  and  timeline. 

•  The  study  should  include  the  CSA/VCSA  expectations  and  requirements  from  the  AoA 
process.  What  is  the  value  of  the  AoA? 

•  Case  studies  selected  need  to  reflect  the  full  spectrum  of  AoAs:  TRAC-led,  AMSAA-led,  ACAT- 
level,  etc. 

•  Which  programs  were  successfully  fielded  and  did  the  AoA  help?  Which  programs  were 
cancelled  and  what  can  we  learn  from  those  AoAs? 


Figure  G-l.  Executive  Summary  Review  from  25  March. 


AoA  PI 


UNCLASSIFIED 


APPENDIX  G  •  57 


UNCLASSIFIED 


CAA-2016058 


Funding: 

•  DAMO-CI  wants  to  clarify  the  governance  process  guiding  the  use  of  AoA  funds  and  desires  a  quarterly  1-  or  2-Star  level  AROC  to 
review  expenditures  to  date  and  discussion  of  the  way  ahead;  others  (TRAC,  AMSAA,  ASA(ALT)  etc.)  want  to  simply  address  funding 
at  the  monthly  prioritization  Council  of  Colonels,  with  any  issues  brought  to  the  1-  or  2-Star  level  as  needed,  but  they  also  said  they 
will  support  whatever  is  necessary  to  get  the  funding  sorted. 

•  DASA-CE  suggested  having  a  plan  in  July  to  defend  the  next  year's  $10M  budget  to  Congress. 

•  Noted  that  all  AoA  funding  would  not  go  to  TRAC  and  AMSAA;  most  would,  but  some  would  go  to  others  (ARL,  SLAD,  etc.).  All 
concurred  with  an  offline  meeting  to  determine  how  to  split  funds  prior  to  the  meeting  with  Mr.  Daniels. 

Other  Discussion  from  EXSUM  slide: 

•  The  study  should  take  on  the  question  of  how  the  Army's  needs  are  different  when  from  CAPE.  The  JROC  is  an  outbrief  of  AoA 
results  (i.e.,  an  information  briefing);  the  AROC  is  a  decision  briefing  informed  by  the  AoA  results  (and  other  factors).  CAPE  is 
looking  for  an  exploration  of  the  trade-space  while  the  components  will  be  looking  at  cross-trades. 

•  Noted  that  the  SAG  does  not  narrow  down  requirements  (as  stated  on  the  EXSUM  slide),  it  confirms  scenarios,  study  issues,  and 
proposed  alternatives. 

•  CAA  noted  that  determining  the  CSA/VCSA  expectations  for  AoAs  will  likely  require  aid  from  LTG  Murray  and  Mr.  Bechtel. 

•  ARCIC  noted  that  ACAT  III  AoAs  are  rare. 

Problem  statement: 

•  In  general,  the  group  preferred  Problem  Statement  2,  but  wanted  to  keep  elements  from  Statement  1  as  objectives  in  the  study. 

•  Problem  Statement  2  should  be  edited  to  read  "that  satisfy  decision-makers'  critical  information  requirements"  instead  of  "that 
satisfy  critical  decision-makers'  information  requirements". 

•  Emphasis  on  making  AoAs  more  effective,  not  just  faster. 

•  A  question  was  raised  whether  we  should  place  in  context  how  the  Army  does  AoAs  vice  the  other  Services.  CAPE  studied  this  in 
2013  and  will  share  their  study. 

•  Need  to  accurately  define  the  purpose  of  an  AoA;  is  it  to  determine  critical  attributes  and  inform  requirements  development,  or  to 
aid  in  acquisition  decisions?  DoDI  5000.02  and  31-70  JCIDS  document  can  help. 

•  PAE  suggested  that  we  add  definitions.  What  are  we  expecting  to  come  into  an  AoA,  are  we  presently  doing  that?  We  need  to 
neck  down  and  lay  out  what  the  expectations  are  all  along  the  way  in  the  process. 

•  Need  to  also  identify  what  we're  expecting  to  be  done  pre-AoA. 


Figure  G-2.  Executive  Summary  from  29  March  (1  of  2). 

Purpose  and  Objectives: 

•  Participants  will  send  new  ideas  based  on  this  meeting's  discussions. 

•  Study  should  include  pre-AoA  activities. 

Literature  Review: 

•  Should  be  sorted  by  the  level  of  the  decision  maker  that  can  effect  a  change  (i.e.,  Congress,  OSD,  Army,  etc.);  additionally,  there 
should  be  an  assumption  of  what  we  can  or  can't  change  (e.g.,  public  law  won't  change) 

•  Also  include  CBA  study  and  CAPE  study. 

Scope: 

•  Noted  that  prioritization  is  the  priority  of  AoAs  relative  to  one  another;  the  AROC  is  how  the  senior  leaders  bless  this 
prioritization. 

•  Don't  talk  just  about  how  the  AROC  was  in  the  past,  but  how  it  could/should  be. 

•  Identify  who  is  the  customer  of  the  AoA  -  Acquisition  Decision  Authority?  JROC?  Congress? 

•  Should  we  put  time  and  resources  towards  items  that  don't  distinguish  between  alternatives? 

•  For  case  studies,  "good"  and  "bad"  are  subjective  terms;  instead  say  process  elements  that  were  good  or  those  that  were 
challenges.  DASA-CE  volunteered  to  provide  some  case  studies  with  cost-related  challenges. 

•  For  case  studies,  there  are  AARs  from  the  AoAs  that  can  help. 

•  Have  to  define  timeline;  large  parts  of  the  overall  AoA  timeline  are  guidance  development  before  the  analysis  and  closeout 
work  after  the  analysis  (e.g.,  admin.,  getting  signatures,  etc.).  Some  parts  we  control,  some  we  don't. 

•  How  should  AoA  Study  Guidance  be  developed?  Guidance  development  is  where  things  are  sometimes  added  just  because 
they're  nice  to  have,  not  because  they  actually  distinguish  between  alternatives;  treated  as  staffing  action.  These  add-ons  can 
increase  the  number  of  alternatives  and  extend  timelines. 

•  How  do  we  define  when  the  AoA  starts  and  is  complete?  Cl  suggested  that  AoA  begins  at  the  start  of  the  1st  SAG;  The  end  is 
driven  when  the  DAB  is  scheduled.  CAPE:  Final  SAG  briefs  do  not  always  embody  what  is  eventually  sent  to  CAPE. 

Recommendations:  The  Decker-Wagner  paper  recommended  requirements  improvements;  nothing  has  been  done  yet. 


Figure  G-3.  Executive  Summary  from  29  March  (2  of  2). 
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Additional  interview  recommendations  suggested:  someone  from  ASA(ALT),  especially  MG  Thurgood  and  the  ASA(ALT);  Dr.  Baker 
(director  of  SLAD)  or  Mr.  Bowen;  Mr.  Amato;  Mr.  Cook  (AEC).  Members  will  help  to  set  up  interviews  as  appropriate. 

Recommended  removing  Crusader  from  the  case  study  list,  as  the  AoA  was  never  actually  completed  before  the  program  was 
cancelled. 

Process  map: 

•  Be  sure  to  include  the  MDD  request  (after  ICD  need,  before  developing  study  guidance);  this  is  a  G-8  function  that  initiates  the 
analytic  efforts  to  prepare  for  the  MDD.  It  identifies  the  lead  agency  for  the  AoA  and  prompts  them  to  start  writing  the  study 
guidance  and  study  plan.  It  indicates  an  interest  in  a  materiel  solution,  though  the  MDD  is  when  the  MDA  actually  decides 
whether  or  not  the  Army  will  pursue  a  materiel  solution.  The  ASARC  is  the  body  that  reviews  the  material  and  makes  the  MDD 
request  (DAB  for  Joint  Systems). 

•  ACAT  III  programs  don't  always  require  all  the  steps  that  ACAT  I  or  II  programs  do,  largely  because  they  often  enter  the 
acquisition  process  pre-MS  B  or  pre-MS  C  (e.g.,  the  technology  may  already  be  mature).  Therefore,  the  first  step  for  an  ACAT  III 
AoA  is  to  look  at  any  existing  analysis  and  see  if  that  satisfies  the  AoA  needs.  Existing  analysis  can  include  analysis  from 
Technology  Demonstrations  and  operational  assessments  from  equipment  used  in  theater,  etc. 

•  Resources  are  limited,  so  leadership  should  identify  what's  most  important. 

•  TRADOC  does  capability  needs  assessment  (CNA),  which  assesses  the  needs  of  the  whole  Army,  not  specific  systems.  They 
don't  have  the  resources  to  do  systems-level  requirements  analysis  on  their  own  for  all  systems;  maybe  the  CoEs  or  RDECs  can 
help  with  that?  Currently  there  are  some  system-level  analyses,  but  they're  at  a  higher  level  and  more  qualitative. 

•  Does  the  Army  have  enough  resources  for  detailed,  quantitative  pre-AoA  analysis?  The  CoEs  are  trying  to  do  this,  previously 
had  limited  analytical  resources  but  getting  better,  but  aren't  quite  there  yet.  ARL/RDECs  are  also  starting  to  get  into  the 
process  earlier,  which  may  help.  Again,  with  limited  resources  would  need  to  prioritize  pre-AoA  analytical  activities. 

•  What  is  the  timeframe  for  the  various  steps  in  the  process  map?  If  there  are  delays,  what  causes  them?  Who  receives  the 
various  outputs/products,  and  how  do  they  use  them?  The  long  poles  in  the  timeline  are  1)  performance  data  development,  2) 
model  development,  especially  combat  models  &  scenario  development  and  3)  life  cycle  cost  analysis.  In  all  three,  the  amount 
of  knowledge  going  in  to  the  process  is  what  dictates  how  long  it  will  take  (e.g.,  are  most  of  the  technologies  well-known,  or  are 
they  new,  meaning  we  don't  have  existing  data  and  models  to  rely  on;  requirement  to  use  new  emerging  scenarios; 
requirement  for  model  modifications  to  replicate  new  concepts  and  behaviors).  How  much  have  we  invested  in  the  previous  FY 
in  data  and  model  development? 


Figure  G-4.  Executive  Summary  from  8  April. 


Challenges/recommendations  to  LTG  Murray  can  be  grouped  into  three  areas: 

•  1)  Pre-AoA  Concepts  and  Decisions. 

•  a)  The  AoA  is  burdened  by  "immature"  ICDs.  The  AoA  must  then  refine  capability  gaps,  requirements, 
and  concepts  of  employment.  This  cannot  be  fixed  with  existing  resources  in  TRADOC. 

•  b)  Army  leadership  does  not  inform  scope  of  the  AoA. 

•  Recommendation:  The  AROC  that  reviews  the  ICD  must  be  a  decision  making  forum  that  isolates 
concepts  for  refinement  and  the  scope  of  the  AoA.  The  AROCM  must  codify  the  decisions  to  constrain 
the  AoA's  scope. 

•  2)  Pre-AoA  Data,  Models,  and  Simulations.  For  developmental  systems,  the  Army  must  develop  data,  scenarios 
and  M&S  to  support  analyses.  When  these  do  not  exist,  a  forecast  of  the  analysis  requirements  is  imperative. 

•  Recommendation:  Some  forum  must  validate  the  AoA  forecast  for  the  analysis  community. 

•  3)  AoA  Process.  Staff  friction,  sequential  reviews,  and  mission  creep  cause  delays  (especially  as  SAGs  don't 
always  have  the  same  people  present,  so  issues  are  brought  up  again  after  already  being  decided). 

•  Recommendation:  Employ  central  coordination  of  AoAs  that  are  planned  and  in  progress.  For  all 
decision-making  forums  (AROCs  and  SAGs),  codify  those  decisions  to  prevent  mission  creep.  Allow 
parallel  reviews  in  TRADOC,  HQDA,  and  CAPE. 

For  the  background  information,  cite  legal  requirements  (e.g.,  WSARA).  (Dr.  Markowitz) 

Cl  currently  "tracks"  the  in-progress  and  planned  AoAs.  They  report  those  biannually  at  the  SAR  and  periodically  with 
TRAC  and  AMSAA.  That  might  not  be  sufficient  to  support  #2  and  #3,  above.  Cl  expressed  that  ASA(ALT)  might  need  t 
weigh  more  heavily  on  that.  (COL  Seaward) 


Figure  G-5.  Executive  Summary  from  12  April  (1  of  2). 
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•  Challenge  #1  must  address  the  "level"  of  analysis.  Some  programs  are  more  complex  and/or  more  important  than 
others.  (We  capture  importance  via  the  ACAT  level.)  Prior  to  the  AoA,  the  scoping  effort  should  reflect  the  complexity 
and  importance  of  the  program  in  question.  (MG  Dyess) 

•  Comment  from  Dr.  Markowitz:  We  need  to  get  VCSA  guidance  sooner.  (Hence,  challenge  lb.) 

•  Comment  from  MG  Richardson  (paraphrased):  I  just  sat  in  an  AROC.  There  was  no  mention  of  the  AoA.  The  VCSA  is 
not  being  given  the  chance  to  influence.  (Again,  challenge  lb). 

•  The  ICD  should  have  an  analytical  annex;  the  requirements  need  rationales  to  properly  define  the  tradespace. 

•  Need  prioritization  of  capability  gaps  and  requirements. 

•  There  is  an  analytical  gap  between  the  CNA  and  the  MDD  request,  and  we  don't  necessarily  have  the  resources  to  do 
this  analysis. 

•  Congress  has  shown  more  interest  recently  in  seeing  AoAs. 

•  The  timeline  of  the  AoA  is  only  a  problem  if  it  delays/holds  up  the  decision  or  the  DAB.  An  AoA  has  never  delayed  a 

DAB. 

•  Resources/workforce  of  the  analytic  community  is  a  problem;  TRAC  currently  has  sufficient  resources,  but  might  not 
in  the  future  as  the  workforce  is  cut  and  workload  increases.  ARCIC's  manpower  to  do  pre-AoA  analysis  is  already  not 
enough  (and  especially  not  trained  enough).  AMSAA's  is  at  bare  bones  now  and  has  targeted  personnel  reductions. 
DAMO-CI  indicated  that  their  projection  for  the  workload  in  late  16/FY17  will  go  up  significantly.  We  might  be  able  to 
take  past  history  data  to  get  a  range  of  past  analytical  capacity  in  order  to  forecast  future  resources,  which  will  be 
something  less  unless  something  changes. 

•  Issues  in  getting  required  data,  especially  foreign  data  or  in  post  MS  A  updates,  where  there's  conflict  with  acquisition 
regulations  and  the  ability  to  get  data  from  contractors.  Also,  do  we  have  access  to  the  right  information  for  pre-AoA 
analysis? 

Figure  G-6.  Executive  Summary  from  12  April  (2  of  2). 
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APPENDIX  H  ARMY  AOA  REVIEW 

H-l  Army  AoA  Duration 


The  average  AoA  duration  for  the  last  10  Anny  AoAs  was  ~13  months,  with  another  2  to  4 
months  for  staffing  and  “sufficiency”  detennination  depending  on  whether  the  program  had 
Anny  or  OSD  oversight. 

H-2  Recent  Army  AoAs  with  CAPE  Oversight 


Armored  Multi-Purpose  Vehicle  (AMPV)  (TRADOC-led),  ACAT  ID.  Short  duration  study  - 
high  quality/detailed  final  report.  All  options  proceed  to  Engineering  and  Manufacturing 
Development  Phase  (EMD)  with  Government  Furnished  Equipment  (GFE)  Hulls  (some  pure  or 
mixed  fleets).  However,  system  subsequently  failed  its  Preliminary  Design  Review  (PDR); 
additional  time  might  have  allowed  for  the  AoA  Study  Team  to  conduct  additional  independent 
requirements/perfonnance  analysis  to  foresee  this  challenge. 

Lower  Tier  Air  and  Missile  Defense  Capability  (LTAMDC)  (TRADOC-Led),  ACAT  ID. 

CAPE  refused  to  schedule  a  SAG  in  August  (too  busy  with  Program  Review).  CAPE  added  a 
late  requirement  to  review  additional  classified  analysis  on  perfonnance  against  an  advanced 
threat  not  included  in  the  study  guidance.  Delay  in  alternative  approval  by  SAG.  Screening 
SAG,  originally  conducted  as  paper  SAG,  did  not  receive  OSD  concurrence.  Screening  was 
re-presented  during  the  next  face-face  SAG.  Alternatives  were  not  finalized  until  analysis  had 
been  completed.  4-star  TRADOC  Staffing  of  final  report.  MDD  conducted  12  December  2014. 

Long-Range  Precision  Fires  (LRPF)  (TRADOC-led),  ACAT  ID.  DAE  expanded  scope  at  the 
MDD  by  adding  requirement  to  address  In-flight  Target  Update  (IFTU)  and  Seeker.  The  office 
of  the  Assistant  Secretary  of  Defense  for  Research  and  Engineering  (ASD(R&E))  requested 
addition  of  a  new  warhead  alternative  during  the  study  that  was  not  competitive  but  required 
effort  to  address  (Kinetic  Energy  Projectile).  CAPE  provided  additional/clarifying  guidance 
during  the  study  that  restricted  the  ability  of  the  Army  to  model  (or  even  represent  the  effects  of) 
low  observable  aircraft  in  the  study  (delayed  completion  of  operational  effectiveness  analysis). 
CAPE  added  a  requirement  to  address  consideration  of  classified  National  Intelligence, 
Surveillance,  and  Reconnaissance  (ISR)  assets  (not  included  in  original  guidance).  4-star 
TRADOC  staffing  of  final  report. 

Improved  Turbine  Engine  Program  (ITEP)  (TRADOC-led),  ACAT  ID,  Pre-MS  A  AoA. 

Perfonnance,  Risk,  Cost,  Affordability  were  the  most  useful  components.  Study  showed 
benefits/weaknesses  of  all  alternatives.  OSD  recommended  Alt  2;  Headquarters,  Department  of 
the  Army  (HQDA)  selected  Alt  4.  ITEP  New  Start  had  been  advertised  as  a  Win-Win  with 
respect  to  Perfonnance  and  Lower  Cost.  Study  showed  that  the  New  Start  produced  greater 
Perfonnance  with  greater  Cost.  Risk  analysis  recommended  changes  to  the  PM  schedule  (which 
were  accepted).  Affordability  analysis  showed  issues  with  funding.  HQDA  changed  priorities  to 
increase  ITEP  funding.  Sustainment  and  reliability  analysis  was  hampered  with  minimal  data  for 
a  pre-MS  A  system.  Differences  between  the  alternatives  were  not  operationally  significant. 
Changes  in  force  structure  were  not  considered.  Delay  in  Anny  completion  of  Life  Cycle  Cost 
Estimates  impacted  study  completion  (-3  months). 
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Joint  Air-to-Ground  Missile  (JAGM)  Increment  I  Update  (TRADOC-led),  ACAT  ID.  The 

initial  intent  in  Defense  Acquisition  Executive  (DAE)  Acquisition  Decision  Memorandum 
(ADM)  and  CAPE  Guidance  was  to  update  Life  Cycle  Costs.  CAPE  required  more  perfonnance 
analysis  than  anticipated.  CAPE  required  the  Anny  to  include  a  foreign  alternative  for  which  the 
Anny  was  unable  to  obtain  detailed  data.  After  significant  effort  to  obtain  performance  data  on 
the  foreign  system,  the  Army  eventually  surrogated  the  performance  (incurring  some  risk  of 
misrepresenting  perfonnance). 

Indirect  Fire  Protection  Capability  IFPC)  Increment  2  (TRADOC-led),  ACAT  1C.  Near 
the  end  of  the  study,  CAPE  directed  a  follow-on  excursion  to  address  an  additional  threat, 
requiring  2  additional  months  to  complete. 

Ground  Combat  Vehicle  (GCV)  (TRADOC-led),  ACAT  1C.  A  high  quality/detailed  MS  A 
final  report  that  enabled  subsequent  Anny  senior  leader  decisions  for  this  system.  AoA  helped 
inform  deferred  investment  decision  and  S&T  work. 

H-3  Recent  Army  AoAs  with  Army  Oversight 


Brownout  Rotor  Craft  Enhancement  System  (BORES)  (AMSAA-led),  ACAT  II.  Planned  as 
MS  A  Decision  AoA,  the  program  was  unfunded  so  the  AoA  continued  with  zero  affordability. 
Significant  initial  delay  in  getting  to  MDD  (ASA(ALT)  adjusted  the  scope  of  the  program  from 
ACAT  I  Degraded  Visual  Environment  (DVE)  Solution  to  ACAT  II  BORES)  and  starting  the 
study  later,  after  the  MDD,  to  allow  HQDA  to  refine  the  study  guidance.  However,  the  Army 
completed  the  study  nearly  on  schedule;  the  affordability  goal  was  $0  since  the  system  was  not 
funded  in  the  POM;  subsequently,  the  Anny  has  funded  this  system. 

H-47  Block  II  (AMSAA-led),  ACAT  1C.  No  significant  issues  -  The  Army  completed  the 
study  on  schedule  in  accordance  with  the  (originally  issued)  CAPE  Guidance  and  HQDA 
Directive.  DAE  delegated  MDA  to  Anny  during  the  study,  and  CAPE  later  relinquished 
oversight  of  the  AoA.  AoA  supported  Request  for  Proposal  (RFP)  release  in  2Q  FY  2016  in 
support  of  a  MS  B  in  2Q  FY  2017.  Cost  and  Affordability  received  the  most  attention  at  the 
Anny  Systems  Acquisition  Review  Council  (ASARC)  and  were  the  most  useful  components  of 
the  AoA.  Production  rates  below  12  were  found  to  be  unsustainable. 

Biometric  Enabling  Capability  (BEC)  Increment  1  AoA  Update  (AMSAA-led),  ACAT  1A. 

Study  was  conducted  to  support  an  Acquisition  strategy  path  forward  and  to  inform  a  cost 
effective,  preferred  alternative  decision  by  the  AAE.  Near  the  end  of  the  study,  emergent  results 
revealed  that  all  alternatives  under  consideration  were  unaffordable.  HQDA  added  additional 
alternatives  for  analysis  that  extended  the  study  by  3  months  but  produced  a  selection  of 
affordable  choices. 

H-4  Key  Pre-MDD  Activities 


Front  End  Analysis  (literature  research,  review  and  prioritize  gaps,  identify  and  define  potential 
alternative  candidates,  and  attribute  development). 

Measurement  Space  Workshop  (Identify  metrics,  define  environmental  and  operational 
conditions,  threat  and  scenario/vignettes,  methods,  models,  and  tools. 

Material  Development  Decision  (MDD)  Brief  Development. 
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Develop  Army  input  for  OSD  CAPE  Study  Guidance. 

OSD  CAPE  issues'  Study  Guidance  /  HQDA  issues'  Study  Directive. 
Study  Plan  Development. 


AoA  PI 


UNCLASSIFIED 


APPENDIX  H  •  63 


CAA-2016058 


UNCLASSIFIED 


(THIS  PAGE  INTENTIONALLY  LEFT  BLANK) 


64  •  APPENDIX  H 


UNCLASSIFIED 


AoA  PI 


